<p>The popularity of short videos has made sentiment analysis crucial for gaining insights into individual attitudes and guiding public opinion. Consequently, Multimodal Sentiment Analysis (MSA) has garnered significant attention in the field of human-computer interaction. Although existing research has made progress, limitations remain: current methods fail to effectively handle redundant information within multimodal data and redundant information generated through inter-modal interactions. This leads to conflicts between modalities and restricts the expressive power of sentiment analysis. To address this, this paper proposes an innovative Multi-Level Perceptual Cross-modal Fusion (MPCF) framework. Through its three-level perceptual design, MPCF introduces novel solutions in three key aspects: pairwise modality fusion, collaborative perception of full multimodal information, and deep interaction of affective information. Specifically, during the pairwise fusion stage, a novel structure is designed that integrates three unique modules. These modules precisely extract latent emotional cues from each modality while effectively eliminating redundant information within them. Simultaneously, an introduced Affective Information Integration Module significantly enhances the collaborative representation of cross-modal features, suppresses the generation of redundant information between modalities, and substantially improves fine-grained perception of affective information across different modalities. Experimental results demonstrate that MPCF achieves excellent performance on three public datasets: CMU-MOSI, CMU-MOSEI, and CH-SIMS, showing particular robustness and high efficiency in sentiment classification tasks.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Multi-level perception cross-modal fusion framework for multimodal sentiment analysis

  • Lei Yu,
  • AnZhan Liu

摘要

The popularity of short videos has made sentiment analysis crucial for gaining insights into individual attitudes and guiding public opinion. Consequently, Multimodal Sentiment Analysis (MSA) has garnered significant attention in the field of human-computer interaction. Although existing research has made progress, limitations remain: current methods fail to effectively handle redundant information within multimodal data and redundant information generated through inter-modal interactions. This leads to conflicts between modalities and restricts the expressive power of sentiment analysis. To address this, this paper proposes an innovative Multi-Level Perceptual Cross-modal Fusion (MPCF) framework. Through its three-level perceptual design, MPCF introduces novel solutions in three key aspects: pairwise modality fusion, collaborative perception of full multimodal information, and deep interaction of affective information. Specifically, during the pairwise fusion stage, a novel structure is designed that integrates three unique modules. These modules precisely extract latent emotional cues from each modality while effectively eliminating redundant information within them. Simultaneously, an introduced Affective Information Integration Module significantly enhances the collaborative representation of cross-modal features, suppresses the generation of redundant information between modalities, and substantially improves fine-grained perception of affective information across different modalities. Experimental results demonstrate that MPCF achieves excellent performance on three public datasets: CMU-MOSI, CMU-MOSEI, and CH-SIMS, showing particular robustness and high efficiency in sentiment classification tasks.