Unified Multimodal Retrieval Framework for Multimodal RAG

Retrieval-Augmented Generation (RAG) mitigates hallucination in large language models but remains limited for text–image mixed documents due to modality separation, OCR-induced semantic fragmentation, and cross-modal similarity inconsistency. We present a unified multimodal retrieval framework that features three key innovations: a unified multimodal encoder to eliminate modality barriers, a post-encoding residual fusion mechanism to preserve unimodal consistency while capturing cross-modal interactions, and a scaled training strategy to correct modality learning imbalance. This framework maps text, images, and text–image units into a shared semantic space, enabling direct retrieval over coherent multimodal chunks with text queries. Experiments on six benchmarks (ArxivQA, ChartQA, DocVQA, InfoVQA, PlotQA, SlideVQA) show state-of-the-art results: our 3.4B model achieves an average +3.99/+4.27 absolute points in Recall@1/Recall@3 over the strongest baseline (GME) and outperforms larger alternatives (e.g., 8.4B E5-V), validating that unified encoding and scaled training effectively alleviate modality separation, similarity inconsistency, and system complexity for multimodal RAG.