Scholarlink

<p>With the rapid development of AI-generated content, enabling machines to accurately infer and recognize human emotions has become crucial for applications such as public opinion monitoring, product optimization, and enhancing user experience. However, existing multimodal sentiment analysis methods predominantly rely on unimodal pre-trained models (e.g., BERT for textual features and ResNet for visual features) and perform feature fusion using simple concatenation or weighted summation. Such approaches fail to effectively model the correlations between modalities, often leading to conflicts in sentiment decisions across different modalities, which ultimately impacts overall performance. To address these limitations, this paper proposes a multi-granularity feature gated fusion network (MFGFN) that combines unimodal and multimodal pre-trained models. The network incorporates a co-attention encoder interaction module to integrate fine-grained unimodal features and coarse-grained cross-modal features, thereby enhancing semantic alignment across modalities. A gated fusion mechanism is introduced to adjust information distribution through similarity-weighted operations, enabling cross-modal interaction and cross-granularity feature integration. Additionally, a mutual information maximization mechanism is employed to enhance information sharing while preserving modality-specific correlations during the fusion process. Experimental results demonstrate that MFGFN significantly outperforms existing methods across multiple datasets. On the MVSA-Single dataset, MFGFN achieves improvements of 0.89% and 1.92% in accuracy and F1-score compared to the best baseline, respectively. Furthermore, it achieves F1-score gains of 0.11% and 0.06% on the MVSA-Multiple and MMSD datasets, respectively, further validating its effectiveness and generalizability in multimodal sentiment analysis tasks.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Multi-grained feature gating fusion network for multimodal sentiment analysis

  • 期刊论文

  • 发表日期：

    2025年4月29日

查看全文
  • Bengong Yu,
  • Chenyue Li,
  • Zhongyu Shi

摘要

With the rapid development of AI-generated content, enabling machines to accurately infer and recognize human emotions has become crucial for applications such as public opinion monitoring, product optimization, and enhancing user experience. However, existing multimodal sentiment analysis methods predominantly rely on unimodal pre-trained models (e.g., BERT for textual features and ResNet for visual features) and perform feature fusion using simple concatenation or weighted summation. Such approaches fail to effectively model the correlations between modalities, often leading to conflicts in sentiment decisions across different modalities, which ultimately impacts overall performance. To address these limitations, this paper proposes a multi-granularity feature gated fusion network (MFGFN) that combines unimodal and multimodal pre-trained models. The network incorporates a co-attention encoder interaction module to integrate fine-grained unimodal features and coarse-grained cross-modal features, thereby enhancing semantic alignment across modalities. A gated fusion mechanism is introduced to adjust information distribution through similarity-weighted operations, enabling cross-modal interaction and cross-granularity feature integration. Additionally, a mutual information maximization mechanism is employed to enhance information sharing while preserving modality-specific correlations during the fusion process. Experimental results demonstrate that MFGFN significantly outperforms existing methods across multiple datasets. On the MVSA-Single dataset, MFGFN achieves improvements of 0.89% and 1.92% in accuracy and F1-score compared to the best baseline, respectively. Furthermore, it achieves F1-score gains of 0.11% and 0.06% on the MVSA-Multiple and MMSD datasets, respectively, further validating its effectiveness and generalizability in multimodal sentiment analysis tasks.