Scholarlink

Zero-shot Visual Question Answering (VQA) is a challenging task that requires intricate reasoning across modalities. Traditional approaches utilizing a single rationale within the Chain of Thoughts (CoT) framework may inadequately address the complex demands of VQA problems. Meanwhile, methods employing multiple rationales frequently suffer from low diversity, suboptimal modality alignment, and inefficient retrieval and fusion. To overcome these limitations, we introduce a novel multi-modal reasoning framework, Mixture of Rationales (MoR). MoR leverages a single, Vision-and-Language Pre-trained Model (VLPM) to dynamically generate, retrieve, and fuse diverse multi-modal rationales, thereby enriching the decision-making process. We evaluate MoR on two challenging VQA datasets, i.e., NLVR2 and OKVQA, with two representative backbones OFA and VL-T5. MoR demonstrates significant improvement with accuracy increase of 12.43% on NLVR2 and 2.45% on the science and technology category of OKVQA (OKVQA-S). Additionally, we have made our implementation code publicly accessible. ( https://github.com/litaohz/mor/tree/0608v1 )

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Mixture of Rationale: Multi-modal Reasoning Mixture for Visual Question Answering

  • 会议论文

  • 发表日期：

    2025年

查看全文
  • Tao Li,
  • Linjun Shou,
  • Xuejun Liu

摘要

Zero-shot Visual Question Answering (VQA) is a challenging task that requires intricate reasoning across modalities. Traditional approaches utilizing a single rationale within the Chain of Thoughts (CoT) framework may inadequately address the complex demands of VQA problems. Meanwhile, methods employing multiple rationales frequently suffer from low diversity, suboptimal modality alignment, and inefficient retrieval and fusion. To overcome these limitations, we introduce a novel multi-modal reasoning framework, Mixture of Rationales (MoR). MoR leverages a single, Vision-and-Language Pre-trained Model (VLPM) to dynamically generate, retrieve, and fuse diverse multi-modal rationales, thereby enriching the decision-making process. We evaluate MoR on two challenging VQA datasets, i.e., NLVR2 and OKVQA, with two representative backbones OFA and VL-T5. MoR demonstrates significant improvement with accuracy increase of 12.43% on NLVR2 and 2.45% on the science and technology category of OKVQA (OKVQA-S). Additionally, we have made our implementation code publicly accessible. ( https://github.com/litaohz/mor/tree/0608v1 )