Scholarlink

<p>Constructing Multimodal Emotion Recognition in Conversation (MERC) models is important for understanding affective states from text, audio, images, and video. Existing approaches often rely on linear layers for cross-modal feature alignment and freeze encoder parameters during training, which can introduce feature degradation and modality misalignment. Conventional one-dimensional positional encoding is also limited when modeling visual and temporal information. To address these issues, this study presents Emotion-BIND, a multimodal framework that integrates ImageBind-based representations with a differentiated positional encoding strategy, termed m-ROPE. The framework maps features from multiple modalities into a shared embedding space and applies 2D RoPE to image representations and 3D RoPE to video representations, supporting multimodal spatial and temporal modeling. Experimental results show that Emotion-BIND achieves an Unweighted Average Recall (UAR) of 49.44%, a Weighted Average Recall (WAR) of 71.00%, and an F1 score of 90.05% on the DFEW dataset, with the highest WAR and F1 score among the compared methods, although its UAR remains lower than the best comparison model. On the MELD dataset, Emotion-BIND achieves a UAR of 55.67%, a WAR of 67.88%, and an F1 score of 91.02%. The source code is archived at <a href="https://doi.org/10.5281/zenodo.21186484">https://doi.org/10.5281/zenodo.21186484</a> and is also available on GitHub at <a href="https://github.com/qunlitang/Emotion-BIND">https://github.com/qunlitang/Emotion-BIND</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Emotion-BIND for multimodal emotion recognition and reasoning

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年7月28日

查看全文
  • Qunli Tang,
  • Qi Wang,
  • Shouhao Zhang,
  • Dongliang Huang

摘要

Constructing Multimodal Emotion Recognition in Conversation (MERC) models is important for understanding affective states from text, audio, images, and video. Existing approaches often rely on linear layers for cross-modal feature alignment and freeze encoder parameters during training, which can introduce feature degradation and modality misalignment. Conventional one-dimensional positional encoding is also limited when modeling visual and temporal information. To address these issues, this study presents Emotion-BIND, a multimodal framework that integrates ImageBind-based representations with a differentiated positional encoding strategy, termed m-ROPE. The framework maps features from multiple modalities into a shared embedding space and applies 2D RoPE to image representations and 3D RoPE to video representations, supporting multimodal spatial and temporal modeling. Experimental results show that Emotion-BIND achieves an Unweighted Average Recall (UAR) of 49.44%, a Weighted Average Recall (WAR) of 71.00%, and an F1 score of 90.05% on the DFEW dataset, with the highest WAR and F1 score among the compared methods, although its UAR remains lower than the best comparison model. On the MELD dataset, Emotion-BIND achieves a UAR of 55.67%, a WAR of 67.88%, and an F1 score of 91.02%. The source code is archived at https://doi.org/10.5281/zenodo.21186484 and is also available on GitHub at https://github.com/qunlitang/Emotion-BIND.