DMBL: Dual-Stage Multimodal Balanced Learning for Multimodal Sentiment Analysis
摘要
Multimodal Sentiment Analysis (MSA) leverages different data sources (e.g. language, audio, and visual) to predict human emotions and has broad applications. However, (a) existing methods often prioritize one modality (typically language) as dominant, which may cause models favoring certain modality over others. (b) the learning state of unimodal features is typically assessed with discrete labels, overlooking the continuity of the feature space. To alleviate these, we propose a Dual-Stage Multimodal Balanced Learning (DMBL) method for MSA, which comprises two components: Bi-perspective Modality Status Appraisal (BMSA) and Multimodal Balanced Fusion (MBF) module. BMSA assesses each modality’s learning status based on discrete labels and contiguous spaces, then reinitializes the unimodal encoder accordingly. MBF dynamically leverages learnable representations to improve non-dominant modalities, enhancing their support for the dominant modality in cross-modal fusion. Extensive experiments on two MSA benchmark datasets show that our approach not only achieves state-of-the-art performance but also promotes balanced learning across different representations. Our code will be open-sourced at https://github.com/TouchYourYearn/DMBL .