<p>The core task of Medical Visual Question Answering (Med-VQA) is to generate accurate answers to clinical questions based on medical images. However, existing models often struggle to simultaneously extract fine-grained details and global semantics due to their reliance on single-scale feature processing. In addition, most methods model the interaction between questions and visual features using bilinear attention, failing to fully leverage visual information to enhance text representations, which leads to suboptimal cross-modal fusion. To address these issues, we propose a novel multi-scale and cross-modal enhancement framework for Med-VQA. Our approach introduces two key innovations: (1) A multi-scale visual encoder that combines vision transformer (ViT) and convolutional neural network (CNN) features through a global–local fusion mechanism, enabling the model to simultaneously capture fine-grained details and global semantics, overcoming the limitations of single-scale processing. (2) A question enhancement module that dynamically refines text representations using visual information before multimodal fusion, enabling more effective cross-modal interactions. By jointly optimizing multi-scale visual feature fusion and text enhancement, our approach more comprehensively integrates image details and global information, effectively capturing cross-modal interactions. This enables the model to better understand the complex relationships between images and text, thereby improving reasoning accuracy. Extensive experiments on the VQA-RAD and SLAKE benchmark datasets show that our model achieves an accuracy of 77.2% on the VQA-RAD dataset and 83.1% on the SLAKE dataset.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A medical visual question-answering model based on multi-scale feature fusion and question Feature enhancement

  • Hong Xia,
  • Yifan Zhang,
  • Hui Jia,
  • Yanping Chen,
  • Jing Xu,
  • Shiyong Li

摘要

The core task of Medical Visual Question Answering (Med-VQA) is to generate accurate answers to clinical questions based on medical images. However, existing models often struggle to simultaneously extract fine-grained details and global semantics due to their reliance on single-scale feature processing. In addition, most methods model the interaction between questions and visual features using bilinear attention, failing to fully leverage visual information to enhance text representations, which leads to suboptimal cross-modal fusion. To address these issues, we propose a novel multi-scale and cross-modal enhancement framework for Med-VQA. Our approach introduces two key innovations: (1) A multi-scale visual encoder that combines vision transformer (ViT) and convolutional neural network (CNN) features through a global–local fusion mechanism, enabling the model to simultaneously capture fine-grained details and global semantics, overcoming the limitations of single-scale processing. (2) A question enhancement module that dynamically refines text representations using visual information before multimodal fusion, enabling more effective cross-modal interactions. By jointly optimizing multi-scale visual feature fusion and text enhancement, our approach more comprehensively integrates image details and global information, effectively capturing cross-modal interactions. This enables the model to better understand the complex relationships between images and text, thereby improving reasoning accuracy. Extensive experiments on the VQA-RAD and SLAKE benchmark datasets show that our model achieves an accuracy of 77.2% on the VQA-RAD dataset and 83.1% on the SLAKE dataset.