Emotion Recognition in Conversations (ERCs) is a vital area within multimodal interaction research, aiming to accurately identify and classify emotions expressed by speakers. Traditional ERC approaches mainly rely on unimodal cues—such as text, audio, or visual data—leading to limited performance. These methods face two key challenges: 1) Consistency in Multimodal Information. Effective integration requires well-aligned and coherent data across modalities. 2) Contextual Information Capture. Capturing evolving emotional dynamics in long conversations is essential. To tackle these issues, we propose a novel Mamba-enhanced Text-Audio-Video alignment network (MaTAV) for the ERC task. MaTAV aligns unimodal features to ensure cross-modal consistency and effectively handles long input sequences to better model contextual emotional flow. Extensive experiments on the MELD and IEMOCAP datasets demonstrate that MaTAV significantly outperforms existing state-of-the-art methods on the ERC task.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Mamba-Enhanced Text-Audio-Video Alignment Network for Emotion Recognition in Conversations

  • Xinran Li,
  • Xiaomao Fan,
  • Qingyang Wu,
  • Xiaojiang Peng,
  • Ye Li

摘要

Emotion Recognition in Conversations (ERCs) is a vital area within multimodal interaction research, aiming to accurately identify and classify emotions expressed by speakers. Traditional ERC approaches mainly rely on unimodal cues—such as text, audio, or visual data—leading to limited performance. These methods face two key challenges: 1) Consistency in Multimodal Information. Effective integration requires well-aligned and coherent data across modalities. 2) Contextual Information Capture. Capturing evolving emotional dynamics in long conversations is essential. To tackle these issues, we propose a novel Mamba-enhanced Text-Audio-Video alignment network (MaTAV) for the ERC task. MaTAV aligns unimodal features to ensure cross-modal consistency and effectively handles long input sequences to better model contextual emotional flow. Extensive experiments on the MELD and IEMOCAP datasets demonstrate that MaTAV significantly outperforms existing state-of-the-art methods on the ERC task.