Confidence-guided dynamic sequential fusion for multimodal sentiment analysis
摘要
Multimodal Sentiment Analysis (MSA) integrates information from different modalities to identify human sentiment. Existing methods often adopt a text-dominant strategy, treat all modal contributions equally, or utilize dynamic fusion, but they often ignore the importance of the sequence of modality fusion, making the fusion process susceptible to the influence of low-quality modalities. To address this problem, we propose a Confidence-guided Dynamic Sequential Fusion (CoDSF) framework for multimodal sentiment analysis. CoDSF estimates modal quality through confidence and selects the best fusion sequence based on the quality of the modality in each sample. It also dynamically adjusts the contribution of each modality to improve the adaptability of the model in complex scenarios. In addition, we design corresponding feature enhancement strategies to improve the accuracy of confidence estimation. We conduct extensive experiments on the CMU-MOSI and CMU-MOSEI datasets. The results show that CoDSF outperforms existing advanced methods in most indicators, effectively alleviates the adverse effects of low-quality modalities, and verifies the effectiveness of the proposed method.