Scholarlink

<p>Medical visual question answering (MVQA) plays a vital role in understanding medical images. However, although foundation models including large language model (LLM) and multimodal model trained on extensive medical datasets have achieved remarkable results in MVQA, they still face significant challenges in multimodal fusion, hallucination problems, and interpretability. To address these issues, we propose KEVQA, a framework for enhancing medical knowledge that integrates foundation model. This framework significantly improves text feature representation by deeply fusing knowledge graph (KG) with multimodal model and language model, and enhances the interpretability of the reasoning process by leveraging path information from KG. Specifically, KEVQA combines KG with multimodal model to strengthen text feature representation and provide interpretability through path information in KG. Moreover, we introduce a knowledge-enhanced prompt engineering method that utilizes structured information from KG to fill knowledge gaps in LLM, enabling effective integration of KG and LLM, and mitigating hallucination phenomena. Experimental results from the VQA-RAD, SLAKE, and PMC-VQA datasets indicate that KEVQA effectively leverages structured knowledge for visual reasoning while greatly improving the reasoning process’s interpretability.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

A knowledge enhanced framework for interpretable medical visual question and answering via large foundation model

  • 期刊论文

  • 发表日期：

    2026年7月11日

查看全文
  • Xinyan Deng,
  • Yinxin Xu,
  • Xiaorou Zheng,
  • Shoubin Dong

摘要

Medical visual question answering (MVQA) plays a vital role in understanding medical images. However, although foundation models including large language model (LLM) and multimodal model trained on extensive medical datasets have achieved remarkable results in MVQA, they still face significant challenges in multimodal fusion, hallucination problems, and interpretability. To address these issues, we propose KEVQA, a framework for enhancing medical knowledge that integrates foundation model. This framework significantly improves text feature representation by deeply fusing knowledge graph (KG) with multimodal model and language model, and enhances the interpretability of the reasoning process by leveraging path information from KG. Specifically, KEVQA combines KG with multimodal model to strengthen text feature representation and provide interpretability through path information in KG. Moreover, we introduce a knowledge-enhanced prompt engineering method that utilizes structured information from KG to fill knowledge gaps in LLM, enabling effective integration of KG and LLM, and mitigating hallucination phenomena. Experimental results from the VQA-RAD, SLAKE, and PMC-VQA datasets indicate that KEVQA effectively leverages structured knowledge for visual reasoning while greatly improving the reasoning process’s interpretability.