Scholarlink

<p>Multimodal learning has demonstrated remarkable performance improvements over unimodal architectures. However, multimodal learning methods often exhibit deteriorated performances if one or more modalities are missing. This may be attributed to the commonly used multi-branch design containing modality-specific components, making such approaches reliant on the availability of a complete set of modalities. In this work, we propose a robust multimodal learning framework, <Emphasis FontCategory="NonProportional">Chameleon</Emphasis>, that adapts a common-space visual learning network to align all input modalities. To enable this, we present the unification of input modalities into one format by encoding any non-visual modality into visual representations thus making it robust to missing modalities. Extensive experiments are performed on multimodal classification task using four textual-visual (Hateful Memes, UPMC Food-101, MM-IMDb, and Ferramenta) and two audio-visual (avMNIST, VoxCeleb) datasets. <Emphasis FontCategory="NonProportional">Chameleon</Emphasis> not only achieves superior performance when all modalities are present at train/test time but also demonstrates notable resilience in the case of missing modalities.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Chameleon: A Multimodal Learning Framework Robust to Missing Modalities

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年6月2日

查看全文
  • Muhammad Irzam Liaqat,
  • Shah Nawaz,
  • Muhammad Zaigham Zaheer,
  • Muhammad Saad Saeed,
  • Hassan Sajjad,
  • Tom De Schepper,
  • Karthik Nandakumar,
  • Muhammad Haris Khan,
  • Ignazio Gallo,
  • Markus Schedl

摘要

Multimodal learning has demonstrated remarkable performance improvements over unimodal architectures. However, multimodal learning methods often exhibit deteriorated performances if one or more modalities are missing. This may be attributed to the commonly used multi-branch design containing modality-specific components, making such approaches reliant on the availability of a complete set of modalities. In this work, we propose a robust multimodal learning framework, Chameleon, that adapts a common-space visual learning network to align all input modalities. To enable this, we present the unification of input modalities into one format by encoding any non-visual modality into visual representations thus making it robust to missing modalities. Extensive experiments are performed on multimodal classification task using four textual-visual (Hateful Memes, UPMC Food-101, MM-IMDb, and Ferramenta) and two audio-visual (avMNIST, VoxCeleb) datasets. Chameleon not only achieves superior performance when all modalities are present at train/test time but also demonstrates notable resilience in the case of missing modalities.