Current Multimodal Large Language Models (MLLMs) lack a structured, cognitively inspired reasoning mechanism for visual emotion understanding, which limits their capacity to distinguish subtle or complex emotions and generalize across domains. To bridge this gap, we propose an instruction-driven visual emotion reasoning approach centered on a novel four-layer hierarchical instruction framework. This framework systematically guides the model through progressive stages of emotion classification, visual interpretation, logical reasoning, and emotion explanation, closely mimicking the human emotional cognitive process. Unlike general-purpose multimodal instruction tuning, our approach is specifically designed for the hierarchical modeling of affective semantics. Our approach integrates MLLMs with a Q-Former fine-tuning mechanism, enabling efficient alignment of visual and emotional semantics while preserving pre-trained knowledge. To tackle the scarcity of affective instruction data, we employ an LLM-based pipeline to automatically construct AffectInstruct-60k, a large-scale dataset comprising 60,879 instruction-response pairs generated through this automated process. Experimental results demonstrate that the proposed method achieves 83.73% recognition accuracy on the EmoSet-118k dataset, surpassing mainstream methods, while showing strong generalization capabilities across domains like Emotion6 and ArtPhoto. The system implementation on the Gradio platform supports image input and emotion reasoning output, validating its potential applications in mental health, intelligent education, and other scenarios.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Emotion Understanding in Visual Scenes via Instruction-Tuned Multimodal Large Language Models

  • Qinglan Wei,
  • Fubin Cao,
  • Mingrui Zhan,
  • Long Ye

摘要

Current Multimodal Large Language Models (MLLMs) lack a structured, cognitively inspired reasoning mechanism for visual emotion understanding, which limits their capacity to distinguish subtle or complex emotions and generalize across domains. To bridge this gap, we propose an instruction-driven visual emotion reasoning approach centered on a novel four-layer hierarchical instruction framework. This framework systematically guides the model through progressive stages of emotion classification, visual interpretation, logical reasoning, and emotion explanation, closely mimicking the human emotional cognitive process. Unlike general-purpose multimodal instruction tuning, our approach is specifically designed for the hierarchical modeling of affective semantics. Our approach integrates MLLMs with a Q-Former fine-tuning mechanism, enabling efficient alignment of visual and emotional semantics while preserving pre-trained knowledge. To tackle the scarcity of affective instruction data, we employ an LLM-based pipeline to automatically construct AffectInstruct-60k, a large-scale dataset comprising 60,879 instruction-response pairs generated through this automated process. Experimental results demonstrate that the proposed method achieves 83.73% recognition accuracy on the EmoSet-118k dataset, surpassing mainstream methods, while showing strong generalization capabilities across domains like Emotion6 and ArtPhoto. The system implementation on the Gradio platform supports image input and emotion reasoning output, validating its potential applications in mental health, intelligent education, and other scenarios.