Scholarlink

<p>Learning from multiple modalities, such as audio and video, offers opportunities for leveraging complementary information, and improving contextual understanding and performance. However, combining such modalities presents challenges, especially when modalities differ in data structure, predictive contribution, and the complexity of their learning processes. It has been observed that one modality can potentially dominate the learning process, hindering the effective utilization of information from other modalities and leading to sub-optimal model performance. To address this issue, the vast majority of previous works suggest assessing the unimodal contributions and dynamically adjusting the training to equalize them. We improve upon previous work by introducing a multi-loss objective and further refining the balancing process, allowing it to dynamically adjust the learning pace of each modality in both directions, acceleration and deceleration, with the ability to phase out balancing effects upon convergence. We validate our approach across five benchmark datasets: CREMA-D for audio-video emotion recognition; AVE and UCF for audio-video action recognition; Something-Something for egocentric action recognition with video and optical flow; and CMU-MOSEI for emotion recognition with audio, video, and text. In addition, we explore alternative methods for quantifying unimodal contributions within multimodal models and analyze the effectiveness of our approach through an examination of modality-specific error distributions across different classes.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Self-Balancing Multimodal Models via Multi-Loss Gradient Modulation

  • 期刊论文

  • 发表日期：

    2026年2月12日

查看全文
  • Konstantinos Kontras,
  • Christos Chatzichristos,
  • Matthew Blaschko,
  • Maarten De Vos

摘要

Learning from multiple modalities, such as audio and video, offers opportunities for leveraging complementary information, and improving contextual understanding and performance. However, combining such modalities presents challenges, especially when modalities differ in data structure, predictive contribution, and the complexity of their learning processes. It has been observed that one modality can potentially dominate the learning process, hindering the effective utilization of information from other modalities and leading to sub-optimal model performance. To address this issue, the vast majority of previous works suggest assessing the unimodal contributions and dynamically adjusting the training to equalize them. We improve upon previous work by introducing a multi-loss objective and further refining the balancing process, allowing it to dynamically adjust the learning pace of each modality in both directions, acceleration and deceleration, with the ability to phase out balancing effects upon convergence. We validate our approach across five benchmark datasets: CREMA-D for audio-video emotion recognition; AVE and UCF for audio-video action recognition; Something-Something for egocentric action recognition with video and optical flow; and CMU-MOSEI for emotion recognition with audio, video, and text. In addition, we explore alternative methods for quantifying unimodal contributions within multimodal models and analyze the effectiveness of our approach through an examination of modality-specific error distributions across different classes.