Scholarlink

<p>Emotion recognition is a crucial aspect of human–computer interaction, enabling intelligent systems to understand and respond to human emotions effectively. However, traditional models struggle with cross-modal integration and lack interpretability, making it challenging to accurately classify emotions from speech, text, and facial expressions. This study proposes an optimized transformer-based multimodal emotion recognition framework trained on the MELD (Multimodal EmotionLines Dataset), which provides real-world dialogue-based emotion annotations. Each modality is processed using specialized deep learning models: BERT/RoBERTa for text to extract context-aware sentence embeddings, wav2vec 2.0 for speech to capture intonation, pitch, and tone variations, and ResNet50/VGG16 for facial expressions to identify emotion-relevant visual features. These extracted representations are fused using a transformer-based cross-modal attention mechanism, and the model undergoes hyperparameter optimization to improve accuracy and generalization. To ensure transparency, SHAP (Shapley additive explanations) is incorporated to identify the most influential features contributing to emotion classification across all modalities. Experimental results demonstrate that the proposed approach outperforms traditional deep learning models on MELD, achieving superior accuracy while maintaining explainability. The findings highlight the potential of integrating explainable AI techniques with deep learning to develop reliable and trustworthy emotion recognition systems.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Interpretable multimodal emotion recognition using optimized transformer model with SHAP-based transparency

  • 期刊论文

  • 发表日期：

    2025年6月17日

查看全文
  • Adel A. Alyoubi,
  • Bader A. Alyoubi

摘要

Emotion recognition is a crucial aspect of human–computer interaction, enabling intelligent systems to understand and respond to human emotions effectively. However, traditional models struggle with cross-modal integration and lack interpretability, making it challenging to accurately classify emotions from speech, text, and facial expressions. This study proposes an optimized transformer-based multimodal emotion recognition framework trained on the MELD (Multimodal EmotionLines Dataset), which provides real-world dialogue-based emotion annotations. Each modality is processed using specialized deep learning models: BERT/RoBERTa for text to extract context-aware sentence embeddings, wav2vec 2.0 for speech to capture intonation, pitch, and tone variations, and ResNet50/VGG16 for facial expressions to identify emotion-relevant visual features. These extracted representations are fused using a transformer-based cross-modal attention mechanism, and the model undergoes hyperparameter optimization to improve accuracy and generalization. To ensure transparency, SHAP (Shapley additive explanations) is incorporated to identify the most influential features contributing to emotion classification across all modalities. Experimental results demonstrate that the proposed approach outperforms traditional deep learning models on MELD, achieving superior accuracy while maintaining explainability. The findings highlight the potential of integrating explainable AI techniques with deep learning to develop reliable and trustworthy emotion recognition systems.