MMKG-RAG: Retrieval-Augmented Generation with Multi-modal Knowledge Graph
摘要
Multi-modal content nowadays has become increasingly prevalent in modern applications. However, most existing RAG-based LLMs still focus on textual data, and fail to understand rich semantic information in multi-modal content. To overcome the issue above, we demonstrate a novel Multi-Modal Knowledge Graph (MMKG)-based RAG system, namely MMKG-RAG. After constructing comprehensive multi-modal knowledge graphs, MMKG-RAG provides the foundation for sophisticated multi-modal content retrieval and question answering capabilities. Compared to the multi-modal LLMs with no MMKG, the developed MMKG-RAG offers more powerful understanding and retrieval, and meanwhile improves its reasoning capability across modalities.