Traditional Retrieval-Augmented Generation (RAG) systems face significant challenges in processing visually-rich, multi-modal documents, especially when addressing complex queries that demand deep semantic understanding and cross-document reasoning. To bridge this gap, we propose a novel framework M3RAG (Multi-hop, Multi-modal, Multi-agent Retrieval-Augmented Generation). M3RAG introduces a collaborative multi-agent architecture that orchestrates an iterative process for query planning, fine-grained multi-modal evidence extraction, and self-correcting answer verification. By dynamically planning information retrieval pathways and fusing cross-modal evidence, M3RAG enables robust reasoning for complex multi-hop questions and achieves a deeper understanding of multi-modal content. Extensive experimental results demonstrate that M3RAG establishes a new state-of-the-art, outperforming existing methods by up to 18.4% on challenging multi-hop, multi-modal document understanding benchmarks, validating its effectiveness in complex reasoning tasks.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

M3RAG: Orchestrating Multi-agent Reasoning for Multi-hop, Multi-modal Understanding

  • Haizhou Du,
  • Wenhao Li

摘要

Traditional Retrieval-Augmented Generation (RAG) systems face significant challenges in processing visually-rich, multi-modal documents, especially when addressing complex queries that demand deep semantic understanding and cross-document reasoning. To bridge this gap, we propose a novel framework M3RAG (Multi-hop, Multi-modal, Multi-agent Retrieval-Augmented Generation). M3RAG introduces a collaborative multi-agent architecture that orchestrates an iterative process for query planning, fine-grained multi-modal evidence extraction, and self-correcting answer verification. By dynamically planning information retrieval pathways and fusing cross-modal evidence, M3RAG enables robust reasoning for complex multi-hop questions and achieves a deeper understanding of multi-modal content. Extensive experimental results demonstrate that M3RAG establishes a new state-of-the-art, outperforming existing methods by up to 18.4% on challenging multi-hop, multi-modal document understanding benchmarks, validating its effectiveness in complex reasoning tasks.