Toward Empathetic AI: Neural-Symbolic LLMs for Emotionally Aligned Conversations
摘要
Large Language Models (LLMs) exhibit impressive capabilities in natural language understanding and generation; however, their ability to interpret and integrate nonverbal emotional cues, such as facial expressions, while maintaining security and interpretability remains underexplored. This study investigates how multimodal LLMs, specifically Qwen2.5-VL and Deepseek-VL, respond to conversational prompts paired with facial expression images through the lens of neural-symbolic integration. We constructed a dataset of 10,000 conversational lines combined with real and synthetic facial expressions depicting various emotional states. Using both automated sentiment analysis and human evaluations based on a 5-point Likert scale, we assessed model responses for tone appropriateness, helpfulness, and emotional alignment. Our results indicate that neural-symbolic integration significantly enhances interpretability and robustness against adversarial inputs, enabling models to achieve a higher average similarity with human interpretations (cosine similarity peaking around 0.7–0.9) compared to Qwen2.5-VL (0.45–0.55). However, both models struggled to accurately interpret subtle or mismatched emotional cues. These findings highlight the potential of neural-symbolic integration to improve the security and emotional reasoning of AI systems.