Scholarlink

In medical video understanding, the m \(^3\) TAGSV task faces challenges in bridging cross-lingual knowledge gaps, modeling cross-modal interactions, and enabling multi-hop reasoning across video segments. To address these, we propose a hierarchical Retrieval-Augmented Generation (RAG)-enhanced tri-modal framework comprising two parallel pathways: (a) a visual predictor using 3D CNNs with temporal shift modules for spatiotemporal feature extraction and (b) a textual predictor processing questions, subtitles, and RAG-augmented knowledge from multilingual medical graphs. The RAG component enriches textual representations via structured triple retrieval, while joint optimization under ground-truth (GT) supervision employs contrastive learning and temporal localization losses to align multimodal features and localize answer segments. A hierarchical attention mechanism aggregates cross-modal and cross-segment evidence for multi-hop reasoning, with cross-modal alignment ensuring precise grounding. Our contributions include a novel tri-modal fusion architecture, an RAG-based multilingual knowledge retrieval mechanism, a hierarchical attention framework for multi-hop reasoning, and a GT-supervised training scheme for accurate temporal answer grounding.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Hierarchical RAG-Driven Multi-hop Reasoning for Medical Video Question Answering

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Ruohan Gao,
  • Qijun Zhao,
  • YangQianQian Chen

摘要

In medical video understanding, the m \(^3\) TAGSV task faces challenges in bridging cross-lingual knowledge gaps, modeling cross-modal interactions, and enabling multi-hop reasoning across video segments. To address these, we propose a hierarchical Retrieval-Augmented Generation (RAG)-enhanced tri-modal framework comprising two parallel pathways: (a) a visual predictor using 3D CNNs with temporal shift modules for spatiotemporal feature extraction and (b) a textual predictor processing questions, subtitles, and RAG-augmented knowledge from multilingual medical graphs. The RAG component enriches textual representations via structured triple retrieval, while joint optimization under ground-truth (GT) supervision employs contrastive learning and temporal localization losses to align multimodal features and localize answer segments. A hierarchical attention mechanism aggregates cross-modal and cross-segment evidence for multi-hop reasoning, with cross-modal alignment ensuring precise grounding. Our contributions include a novel tri-modal fusion architecture, an RAG-based multilingual knowledge retrieval mechanism, a hierarchical attention framework for multi-hop reasoning, and a GT-supervised training scheme for accurate temporal answer grounding.