Multimodal learning has emerged as a pivotal advancement in artificial intelligence by integrating textual and non-textual data to enhance both understanding and generative capabilities. However, the challenge of missing modalities during training often leads to degraded performance, thereby limiting the applicability of these models in real-world scenarios. In this paper, we propose a comprehensive solution to the problem of missing modalities in multimodal learning. Our approach introduces a Modality-Resilient Framework (MRF) that integrates modality-invariant feature learning, cross-modal attention mechanisms, and an advanced imputation strategy that leverages inter-modal relationships. Extensive experiments on benchmark datasets demonstrate that the proposed framework not only outperforms existing methods but also substantially enhances robustness in the presence of incomplete modality inputs.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

MRF: A Modality-Resilient Framework for Handling Missing Modalities in Multimodal Learning

  • Pengfei Du,
  • Yongjun Huang,
  • Xu Chang,
  • Ruifan Li

摘要

Multimodal learning has emerged as a pivotal advancement in artificial intelligence by integrating textual and non-textual data to enhance both understanding and generative capabilities. However, the challenge of missing modalities during training often leads to degraded performance, thereby limiting the applicability of these models in real-world scenarios. In this paper, we propose a comprehensive solution to the problem of missing modalities in multimodal learning. Our approach introduces a Modality-Resilient Framework (MRF) that integrates modality-invariant feature learning, cross-modal attention mechanisms, and an advanced imputation strategy that leverages inter-modal relationships. Extensive experiments on benchmark datasets demonstrate that the proposed framework not only outperforms existing methods but also substantially enhances robustness in the presence of incomplete modality inputs.