IML-CMM - A Multimodal Sentiment Analysis Framework Integrating Intra-modal Learning and Cross-Modal Mixup Enhancement
摘要
Existing multimodal sentiment analysis methods, while effective at extracting high-level unimodal features, still face challenges in the coordinated fusion of cross-modal information. These methods often struggle to fully leverage the complementary nature of different modalities. To address these limitations, this paper proposes a novel multimodal sentiment analysis framework that combines intra-modal feature learning with cross-modal mixup enhancement, termed IML-CMM. The model first utilizes KAN (Kolmogorov-Arnold Network) and Transformer to construct intra-modal feature extraction layers, which progressively embed features from text, audio, and video modalities, capturing key information at each layer. Subsequently, an Adaptive Hyper-Modality Learning (AHL) module adjusts dynamic weights between audio and video modalities, guided by multi-scale textual features, to enhance multimodal fusion. To further optimize cross-modal learning, we introduce an Audio-Visual Mixup Enhancement technique. This method mixes acoustic and visual modalities from different video sources to generate new samples, producing a consistency loss between mixed audiovisual data. The combined loss from both multimodal and mixup consistency losses is used as the overall loss, improving the model’s generalization to diverse emotional cues. Experimental results demonstrate that the proposed model achieves improvements of 1.64% to 11.14% across various metrics on the CH-SIMS v2 dataset, validating the effectiveness of its cross-modal learning