Scholarlink

<p>In multimodal sentiment analysis, effectively utilizing and fusing information from multiple modalities remains a challenging task. Most existing studies focus on single-modal information, neglecting the potential of multimodal data. To address this, we propose a Transformer-based short-term memory attention (S-MA) model that captures both intra- and inter-modal interactions, learns the weight distribution between different modalities, and enhances modality representations. The model introduces a short-term memory attention module to retain significant features obtained from the previous training session, employing Transformer structures for both intra-modal and inter-modal interactions. Additionally, we introduce a self-distillation method that uses early-stage model outputs as soft labels to guide subsequent training, optimizing the model's representational capabilities. Experimental results on three public datasets demonstrate that the S-MA model outperforms previous state-of-the-art baselines, particularly excelling on the MVSA-Single and HFM datasets, with improvements of 1.98, 1.43 and 1.67, 1.75 percentage points in accuracy (ACC) and F1 metrics, respectively. The source code and datasets are available at [<a href="https://github.com/Doyken/S-MA">https://github.com/Doyken/S-MA</a>].</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Transformer-based short-term memory attention for enhanced multimodal sentiment analysis

  • 期刊论文

  • 发表日期：

    2025年4月9日

查看全文
  • Dangguo Shao,
  • Kaiqiang Tang,
  • Jingtao Li,
  • Sanli Yi,
  • Lei Ma

摘要

In multimodal sentiment analysis, effectively utilizing and fusing information from multiple modalities remains a challenging task. Most existing studies focus on single-modal information, neglecting the potential of multimodal data. To address this, we propose a Transformer-based short-term memory attention (S-MA) model that captures both intra- and inter-modal interactions, learns the weight distribution between different modalities, and enhances modality representations. The model introduces a short-term memory attention module to retain significant features obtained from the previous training session, employing Transformer structures for both intra-modal and inter-modal interactions. Additionally, we introduce a self-distillation method that uses early-stage model outputs as soft labels to guide subsequent training, optimizing the model's representational capabilities. Experimental results on three public datasets demonstrate that the S-MA model outperforms previous state-of-the-art baselines, particularly excelling on the MVSA-Single and HFM datasets, with improvements of 1.98, 1.43 and 1.67, 1.75 percentage points in accuracy (ACC) and F1 metrics, respectively. The source code and datasets are available at [https://github.com/Doyken/S-MA].