Multi-hop Knowledge-Enhanced Query Reasoning for Multi-modal Medical Video QA
摘要
Multi-modal medical instructional video question answering (MMI-VQA) requires precise alignment of visual and textual information, along with complex multi-hop reasoning over external knowledge sources. Existing approaches often struggle to effectively integrate multi-modal signals and external knowledge for accurate retrieval and temporal localization. In this paper, we propose a multi-hop knowledge-enhanced cross-modal retrieval framework for NLPCC-2025 Task 4. Our method first encodes subtitles and their temporally aligned video frames through dedicated encoders, and performs cross-modal interaction and temporal modeling to construct a fine-grained retrieval library. Given an input query, relevant knowledge triples are retrieved from an external medical knowledge graph and enhanced through multi-hop reasoning in a Retrieval-Augmented Generation (RAG) module, producing an enriched query representation. The enhanced query vector is then used to compute similarity scores with subtitle segment embeddings in the retrieval library, enabling retrieval of top-k relevant segments and prediction of their temporal spans. Our framework effectively combines external medical knowledge with multi-modal understanding, offering a scalable solution for complex medical instructional video QA scenarios.