Scholarlink

<p>Accurate emotion recognition in dialogue depends on making effective use of conversational context, yet many multimodal systems under-utilize this contextual information and misclassify nuanced speech. We introduce <i>Multimodal Contextual Transformer Augmented Fusion</i> (MCTAF), a lightweight context-sensitive transformer-based multimodal model that encodes speech and transcripts with Bi-GRUs and treats the last <i>K</i> utterances as a separate modality for contextual information. The context module summarizes the most recent <i>K</i> preceding <i>utterances</i> into a single <i>context vector</i>. This vector is fed to the fusion transformer as its own information stream, so cross-attention can relate the current text and audio to dialogue history without simply concatenating features. All modality vectors operate within a shared 128-dimensional space and are integrated through twelve directed cross-modal attention over text, audio, context modality, and an early-fusion feature. Trained end-to-end with Adam, MCTAF attains 89.9% accuracy on IEMOCAP and 88.3% on MELD, boosting weighted F1 by up to 3% and accuracy by up to 4% points over strong state-of-the-art methods. Ablation studies show that removing the context branch lowers weighted&#xa0;F1 by 3 to 4 points, demonstrating the importance of modeling explicit contextual module for improved emotion recognition. These results on benchmark datasets show that the inclusion of a dedicated contextual module yields consistent gains in both dyadic and multi-party conversations with modest computational cost.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Multimodal contextual transformer augmented fusion for emotion recognition

  • 期刊论文

  • 发表日期：

    2025年12月13日

查看全文
  • Wesagn Dawit Chemma,
  • Adane Letta Mamuye,
  • Marco Piangerelli

摘要

Accurate emotion recognition in dialogue depends on making effective use of conversational context, yet many multimodal systems under-utilize this contextual information and misclassify nuanced speech. We introduce Multimodal Contextual Transformer Augmented Fusion (MCTAF), a lightweight context-sensitive transformer-based multimodal model that encodes speech and transcripts with Bi-GRUs and treats the last K utterances as a separate modality for contextual information. The context module summarizes the most recent K preceding utterances into a single context vector. This vector is fed to the fusion transformer as its own information stream, so cross-attention can relate the current text and audio to dialogue history without simply concatenating features. All modality vectors operate within a shared 128-dimensional space and are integrated through twelve directed cross-modal attention over text, audio, context modality, and an early-fusion feature. Trained end-to-end with Adam, MCTAF attains 89.9% accuracy on IEMOCAP and 88.3% on MELD, boosting weighted F1 by up to 3% and accuracy by up to 4% points over strong state-of-the-art methods. Ablation studies show that removing the context branch lowers weighted F1 by 3 to 4 points, demonstrating the importance of modeling explicit contextual module for improved emotion recognition. These results on benchmark datasets show that the inclusion of a dedicated contextual module yields consistent gains in both dyadic and multi-party conversations with modest computational cost.