Multimodal Sentiment Analysis (MSA) leverages different data sources (e.g. language, audio, and visual) to predict human emotions and has broad applications. However, (a) existing methods often prioritize one modality (typically language) as dominant, which may cause models favoring certain modality over others. (b) the learning state of unimodal features is typically assessed with discrete labels, overlooking the continuity of the feature space. To alleviate these, we propose a Dual-Stage Multimodal Balanced Learning (DMBL) method for MSA, which comprises two components: Bi-perspective Modality Status Appraisal (BMSA) and Multimodal Balanced Fusion (MBF) module. BMSA assesses each modality’s learning status based on discrete labels and contiguous spaces, then reinitializes the unimodal encoder accordingly. MBF dynamically leverages learnable representations to improve non-dominant modalities, enhancing their support for the dominant modality in cross-modal fusion. Extensive experiments on two MSA benchmark datasets show that our approach not only achieves state-of-the-art performance but also promotes balanced learning across different representations. Our code will be open-sourced at https://github.com/TouchYourYearn/DMBL .

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

DMBL: Dual-Stage Multimodal Balanced Learning for Multimodal Sentiment Analysis

  • YiYang Tang,
  • Qian Chen,
  • NanJie Zheng,
  • Ning Luo

摘要

Multimodal Sentiment Analysis (MSA) leverages different data sources (e.g. language, audio, and visual) to predict human emotions and has broad applications. However, (a) existing methods often prioritize one modality (typically language) as dominant, which may cause models favoring certain modality over others. (b) the learning state of unimodal features is typically assessed with discrete labels, overlooking the continuity of the feature space. To alleviate these, we propose a Dual-Stage Multimodal Balanced Learning (DMBL) method for MSA, which comprises two components: Bi-perspective Modality Status Appraisal (BMSA) and Multimodal Balanced Fusion (MBF) module. BMSA assesses each modality’s learning status based on discrete labels and contiguous spaces, then reinitializes the unimodal encoder accordingly. MBF dynamically leverages learnable representations to improve non-dominant modalities, enhancing their support for the dominant modality in cross-modal fusion. Extensive experiments on two MSA benchmark datasets show that our approach not only achieves state-of-the-art performance but also promotes balanced learning across different representations. Our code will be open-sourced at https://github.com/TouchYourYearn/DMBL .