Multimodal sentiment analysis (MSA) has garnered increasing attention for its ability to integrate information from diverse modalities, such as text, audio, and video, to improve sentiment prediction. However, modality imbalance and inter-modal incongruity remain significant challenges, as different modalities contribute unequally to sentiment understanding in various contexts. In this paper, we propose a Dynamic Dominate-Modal-aware Model (DDMM) that addresses these issues by adaptively selecting the most informative modality for each input and enhancing cross-modal interactions through a novel attention-based fusion strategy. Our framework consists of three key components: 1) dedicated encoders for each modality (text, audio, video) to extract modality-specific features; 2) a Dynamic Dominant Modality Selection mechanism that computes an importance score for each modality and selects the most relevant one for sentiment analysis; and 3) a Cross-Dominant-Modal Attention Fusion module that integrates the complementary information from auxiliary modalities into the dominant modality’s representation. This adaptive and selective fusion improves the robustness and precision of sentiment predictions. Extensive experiments on benchmark datasets (CMU-MOSI and CMU-MOSEI) demonstrate that our approach effectively mitigates modality imbalance and achieves superior performance in both sentiment classification and regression tasks, outperforming existing methods. Our model provides a robust solution for real-world multimodal sentiment analysis applications, where input modalities vary in quality and relevance.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Dynamic Dominate-Modal-Aware Model for Multimodal Sentiment Analysis

  • Jianing Zhao,
  • Qun Jin

摘要

Multimodal sentiment analysis (MSA) has garnered increasing attention for its ability to integrate information from diverse modalities, such as text, audio, and video, to improve sentiment prediction. However, modality imbalance and inter-modal incongruity remain significant challenges, as different modalities contribute unequally to sentiment understanding in various contexts. In this paper, we propose a Dynamic Dominate-Modal-aware Model (DDMM) that addresses these issues by adaptively selecting the most informative modality for each input and enhancing cross-modal interactions through a novel attention-based fusion strategy. Our framework consists of three key components: 1) dedicated encoders for each modality (text, audio, video) to extract modality-specific features; 2) a Dynamic Dominant Modality Selection mechanism that computes an importance score for each modality and selects the most relevant one for sentiment analysis; and 3) a Cross-Dominant-Modal Attention Fusion module that integrates the complementary information from auxiliary modalities into the dominant modality’s representation. This adaptive and selective fusion improves the robustness and precision of sentiment predictions. Extensive experiments on benchmark datasets (CMU-MOSI and CMU-MOSEI) demonstrate that our approach effectively mitigates modality imbalance and achieves superior performance in both sentiment classification and regression tasks, outperforming existing methods. Our model provides a robust solution for real-world multimodal sentiment analysis applications, where input modalities vary in quality and relevance.