<p>The goal of multimodal sentiment analysis is to efficiently identify and interpret human emotions by integrating multiple modalities (e.g., text, audio, and video). Traditional representation learning techniques often fail to adequately address inter-modal heterogeneity and temporal continuity, particularly as multimodal sentiment analysis tasks grow in complexity. Consequently, these methods struggle to achieve effective cross-modal fusion while mitigating redundant information and noise interference. To address these challenges, we propose DRTSC, a novel multimodal sentiment analysis framework. First, the framework employs disentangled representation learning to extract shared and private features; introduces a temporal smoothness loss to enforce consistency in audio and video features; and incorporates adversarial loss with backward tuning. Second, a textual hierarchical guidance module coordinates audio and video emotional expressions by leveraging affective cues from text. Finally, efficient feature fusion is achieved through cross-modal interaction layers. Extensive experiments on CMU-MOSI and CMU-MOSEI benchmarks demonstrate that the proposed model achieves state-of-the-art performance in sentiment analysis tasks.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Disentangled representation learning with temporal smoothness constraints for multimodal sentiment analysis

  • Yihao Xu,
  • Hai Huan

摘要

The goal of multimodal sentiment analysis is to efficiently identify and interpret human emotions by integrating multiple modalities (e.g., text, audio, and video). Traditional representation learning techniques often fail to adequately address inter-modal heterogeneity and temporal continuity, particularly as multimodal sentiment analysis tasks grow in complexity. Consequently, these methods struggle to achieve effective cross-modal fusion while mitigating redundant information and noise interference. To address these challenges, we propose DRTSC, a novel multimodal sentiment analysis framework. First, the framework employs disentangled representation learning to extract shared and private features; introduces a temporal smoothness loss to enforce consistency in audio and video features; and incorporates adversarial loss with backward tuning. Second, a textual hierarchical guidance module coordinates audio and video emotional expressions by leveraging affective cues from text. Finally, efficient feature fusion is achieved through cross-modal interaction layers. Extensive experiments on CMU-MOSI and CMU-MOSEI benchmarks demonstrate that the proposed model achieves state-of-the-art performance in sentiment analysis tasks.