With the rapid advancement of multimodal large models, there is an increasing demand for intelligent question-answering systems capable of comprehending and leveraging complex, heterogeneous, and multi-source knowledge. However, existing approaches often face challenges in semantic alignment, efficient retrieval, and coherent generation, particularly when processing unstructured multimodal data and multi-turn dialogues. To address these limitations, this study presents a systematically designed multimodal retrieval-augmented generation (RAG) system that integrates advanced knowledge segmentation, hybrid retrieval, and generation techniques. In the offline stage, the system performs fine-grained knowledge segmentation and structured annotation through multimodal understanding, while enhancing global retrieval performance via multi-dimensional knowledge augmentation. In the online stage, it establishes an end-to-end, closed-loop answering pipeline that incorporates multi-turn dialogue rewriting, hybrid retrieval, reranking, and large model generation. Furthermore, a comprehensive automatic evaluation framework is developed to assess the system across multiple dimensions, including retrieval accuracy, factual consistency, generation quality, and structural clarity. Experimental results show that the proposed system substantially improves retrieval hit rates and generation performance over baseline methods. Overall, this work demonstrates the effectiveness of integrating RAG frameworks with multimodal large models and provides a solid foundation for deployment in high-demand domains such as healthcare, finance, and education.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A Multimodal Retrieval-Augmented Generation System for Intelligent Question Answering

  • Bo Liu,
  • Yishuang Ning,
  • Sheng He,
  • Fei Guo,
  • Siyu Jia,
  • Li Zhu

摘要

With the rapid advancement of multimodal large models, there is an increasing demand for intelligent question-answering systems capable of comprehending and leveraging complex, heterogeneous, and multi-source knowledge. However, existing approaches often face challenges in semantic alignment, efficient retrieval, and coherent generation, particularly when processing unstructured multimodal data and multi-turn dialogues. To address these limitations, this study presents a systematically designed multimodal retrieval-augmented generation (RAG) system that integrates advanced knowledge segmentation, hybrid retrieval, and generation techniques. In the offline stage, the system performs fine-grained knowledge segmentation and structured annotation through multimodal understanding, while enhancing global retrieval performance via multi-dimensional knowledge augmentation. In the online stage, it establishes an end-to-end, closed-loop answering pipeline that incorporates multi-turn dialogue rewriting, hybrid retrieval, reranking, and large model generation. Furthermore, a comprehensive automatic evaluation framework is developed to assess the system across multiple dimensions, including retrieval accuracy, factual consistency, generation quality, and structural clarity. Experimental results show that the proposed system substantially improves retrieval hit rates and generation performance over baseline methods. Overall, this work demonstrates the effectiveness of integrating RAG frameworks with multimodal large models and provides a solid foundation for deployment in high-demand domains such as healthcare, finance, and education.