Multi-modal medical instructional video question answering (MMI-VQA) requires precise alignment of visual and textual information, along with complex multi-hop reasoning over external knowledge sources. Existing approaches often struggle to effectively integrate multi-modal signals and external knowledge for accurate retrieval and temporal localization. In this paper, we propose a multi-hop knowledge-enhanced cross-modal retrieval framework for NLPCC-2025 Task 4. Our method first encodes subtitles and their temporally aligned video frames through dedicated encoders, and performs cross-modal interaction and temporal modeling to construct a fine-grained retrieval library. Given an input query, relevant knowledge triples are retrieved from an external medical knowledge graph and enhanced through multi-hop reasoning in a Retrieval-Augmented Generation (RAG) module, producing an enriched query representation. The enhanced query vector is then used to compute similarity scores with subtitle segment embeddings in the retrieval library, enabling retrieval of top-k relevant segments and prediction of their temporal spans. Our framework effectively combines external medical knowledge with multi-modal understanding, offering a scalable solution for complex medical instructional video QA scenarios.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Multi-hop Knowledge-Enhanced Query Reasoning for Multi-modal Medical Video QA

  • Yangchengyu Zhou,
  • Jiayuan Wu,
  • Yunze Li

摘要

Multi-modal medical instructional video question answering (MMI-VQA) requires precise alignment of visual and textual information, along with complex multi-hop reasoning over external knowledge sources. Existing approaches often struggle to effectively integrate multi-modal signals and external knowledge for accurate retrieval and temporal localization. In this paper, we propose a multi-hop knowledge-enhanced cross-modal retrieval framework for NLPCC-2025 Task 4. Our method first encodes subtitles and their temporally aligned video frames through dedicated encoders, and performs cross-modal interaction and temporal modeling to construct a fine-grained retrieval library. Given an input query, relevant knowledge triples are retrieved from an external medical knowledge graph and enhanced through multi-hop reasoning in a Retrieval-Augmented Generation (RAG) module, producing an enriched query representation. The enhanced query vector is then used to compute similarity scores with subtitle segment embeddings in the retrieval library, enabling retrieval of top-k relevant segments and prediction of their temporal spans. Our framework effectively combines external medical knowledge with multi-modal understanding, offering a scalable solution for complex medical instructional video QA scenarios.