Multimodal Sentiment Analysis with Modality-Robust and -Biased Representations and Distance-Aware Contrastive Learning
摘要
Multimodal Sentiment Analysis (MSA) leverages information from multiple modalities to predict sentiment labels. However, existing approaches often suffer from the challenge of fitting spurious correlations between multimodal features and sentiment labels, with varying degrees of spurious correlations across different modalities. To address this issue, we propose MbrCL, a novel framework that integrates dynamic fusion and distance-aware contrastive learning. The dynamic fusion process is carried out in three steps: (1) disentangling robust and biased features within each modality, (2) estimating the degree of bias using the biased features, and (3) determining implicit and dominant modalities based on bias weights. Implicit modalities are employed to guide the computation of inter-modal correlations. Subsequently, the multimodal features are utilized to compute mixed-modal correlations through our proposed distance-aware contrastive learning, which incorporates affective label distance information. Finally, we identify sentiment-related information using a self-attention mechanism based on mixed-modal correlations and robust features extracted from visual and audio modalities. Experimental results on CMU-MOSI and CMU-MOSEI demonstrate that our model achieves state-of-the-art performance by effectively eliminating the interference of spurious correlations, leading to more accurate sentiment predictions.