<p>Multimodal sentiment analysis (MSA) infers human sentiment by jointly modeling textual, auditory, and visual signals and is a key component in intelligent interaction systems. Performance, however, is often degraded by cross-modal noise and insufficiently optimized feature fusion. This study proposes NIAMDF, a model based on noise injection and adaptive multi-view dynamic fusion. To suppress the propagation of cross-modal noise, an adversarial noise suppression module couples noise injection with adversarial training, enabling intervention during feature generation. A collaborative constraint mechanism is additionally imposed, promoting consistency between disentangled representations and the original semantics. For adaptive fine-grained fusion, a multi-view design is adopted: modality-shared dynamic knowledge fusion performs context-adaptive filtering of shared representations, modality-specific cross-modal attention extracts modality-unique affective cues, and multi-expert gating fusion aggregates multi-perspective representations while dynamically adjusting modality contribution weights. Experiments are conducted on three widely used MSA datasets (CMU-MOSI, CMU-MOSEI, and CH-SIMS). NIAMDF achieves competitive performance relative to state-of-the-art models. The model also remains effective in challenging cases involving inconsistent affective expressions across modalities, cross-modal noise interference, and information redundancy.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Multimodal sentiment analysis based on noise injection and adaptive multi-view dynamic fusion

  • Wei Zheng,
  • Qingtao Chen,
  • Zhenlin Zhang,
  • Caifeng Cui

摘要

Multimodal sentiment analysis (MSA) infers human sentiment by jointly modeling textual, auditory, and visual signals and is a key component in intelligent interaction systems. Performance, however, is often degraded by cross-modal noise and insufficiently optimized feature fusion. This study proposes NIAMDF, a model based on noise injection and adaptive multi-view dynamic fusion. To suppress the propagation of cross-modal noise, an adversarial noise suppression module couples noise injection with adversarial training, enabling intervention during feature generation. A collaborative constraint mechanism is additionally imposed, promoting consistency between disentangled representations and the original semantics. For adaptive fine-grained fusion, a multi-view design is adopted: modality-shared dynamic knowledge fusion performs context-adaptive filtering of shared representations, modality-specific cross-modal attention extracts modality-unique affective cues, and multi-expert gating fusion aggregates multi-perspective representations while dynamically adjusting modality contribution weights. Experiments are conducted on three widely used MSA datasets (CMU-MOSI, CMU-MOSEI, and CH-SIMS). NIAMDF achieves competitive performance relative to state-of-the-art models. The model also remains effective in challenging cases involving inconsistent affective expressions across modalities, cross-modal noise interference, and information redundancy.