Multimodal sentiment analysis aims to integrate information from textual, acoustic, and visual modalities to improve emotional understanding. However, existing methods still face challenges in handling feature redundancy and achieving effective cross-modal representation learning. In this paper, we propose MAGI (Modality-Aligned Geometry-aware Integration), a novel framework that explicitly disentangles multimodal inputs into modality-shared and modality-specific representations for more robust modeling. MAGI employs a shared encoder and modality-specific encoders to perform representation disentanglement, guided by geometry-aware regularization losses including reconstruction, specificity, and soft orthogonality constraints. To align the shared representations across modalities, we introduce a volume-based contrastive loss that encourages geometric consistency in the latent space. Meanwhile, the modality-specific representations are enhanced via a cross-modal attention mechanism and then fused using a gating module that adaptively integrates modality-aware information. Finally, MAGI incorporates a triple-space prediction architecture that jointly utilizes shared, specific, and fused representations to improve prediction accuracy. Experimental results on CMU-MOSI and CMU-MOSEI demonstrate that MAGI achieves competitive performance across multiple evaluation metrics. Further ablation studies confirm the effectiveness of each core component.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

MAGI: Modality-Aligned Geometry-Aware Integration for Robust Multimodal Sentiment Analysis

  • Bohan Hu,
  • Yuhang Li

摘要

Multimodal sentiment analysis aims to integrate information from textual, acoustic, and visual modalities to improve emotional understanding. However, existing methods still face challenges in handling feature redundancy and achieving effective cross-modal representation learning. In this paper, we propose MAGI (Modality-Aligned Geometry-aware Integration), a novel framework that explicitly disentangles multimodal inputs into modality-shared and modality-specific representations for more robust modeling. MAGI employs a shared encoder and modality-specific encoders to perform representation disentanglement, guided by geometry-aware regularization losses including reconstruction, specificity, and soft orthogonality constraints. To align the shared representations across modalities, we introduce a volume-based contrastive loss that encourages geometric consistency in the latent space. Meanwhile, the modality-specific representations are enhanced via a cross-modal attention mechanism and then fused using a gating module that adaptively integrates modality-aware information. Finally, MAGI incorporates a triple-space prediction architecture that jointly utilizes shared, specific, and fused representations to improve prediction accuracy. Experimental results on CMU-MOSI and CMU-MOSEI demonstrate that MAGI achieves competitive performance across multiple evaluation metrics. Further ablation studies confirm the effectiveness of each core component.