Multi-level language interaction transformer for multimodal sentiment analysis
摘要
Multimodal sentiment analysis is a rapidly evolving research field that combines data from various modalities such as text, images, and sound to analyze and understand human emotional states. This approach comprehensively captures the nuances of emotional expression, as people often convey emotions in multiple ways during communication, such as text, tone, facial expressions, and body language. However, redundant information between different modalities and irrelevant information unique to a single modality may hinder further improvement in model performance during cross-modality fusion. To address this, we propose a multimodal sentiment analysis method with multi-level language interaction. This method uses language features extracted at different scales by multi-layer Transformers combined with multimodal interaction layers to interact with visual and audio modalities at various levels, extracting complementary information between modalities and reducing redundancy and irrelevant interference information in the modality fusion process. Finally, the interaction’s feature representation is used for sentiment prediction. Experiments show that our model has achieved the best performance on several common datasets (MOSI, MOSEI, and CH-SIMS), especially with significant improvements in more fine-grained classification (Acc-5 and Acc-7).