Text-dominant strategy for multistage optimized modality fusion in multimodal sentiment analysis

In the current multimodal sentiment analysis, multimodal data fusion has gradually transitioned from non-modality-dominant to modality-dominant. Effectively realizing the fusion of relevant information between different modalities is still one of the crucial challenges facing multimodal sentiment analysis. Although existing multimodal fusion methods have made some progress, most methods still show limitations in dealing with the complex dynamic relationships between modalities. This study proposes a text-dominant Multistage Modality Fusion (DMMF) mechanism. In the first stage, the text modalities are guided through the auxiliary modalities of vision and audio modalities to perform auxiliary modal fusion at the embedding layer. In the second stage, the extracted features can be utilized to enhance the multimodal interactions between different modalities through the mechanisms of self-attention and cross-attention to develop the fusion and achieve better fusion representation and fusion effect in multistage. The model is comprehensively evaluated on the CMU-MOSI and CMU-MOSEI datasets. The experimental results show a significant improvement in most evaluation metrics of our model compared to the baseline models. The code is available on https://github.com/mmm587/DMMF.