Scholarlink

The increasing need for the integration of different kinds of medical data, especially images and clinical documentation, is necessary for improving diagnosis, clinical decision-making, and the delivery of healthcare to patients. We are developing a Multimodal Medical AI Assistant to extract the pertinent information from a variety of medical imaging modalities using state-of-the-art Vision-Language Models such as BLIP-2, InstructBLIP, BioMedCLIP, OpenCLIP, and ViLT. We are evaluating our Multimodal Medical AI Assistant using established benchmark datasets like VQA-RAD, SLAKE, OmniMedVQA, and Medical Multimodal Evaluation Dataset. The results reveal that BioMedCLIP consistently produces the best results across all modalities with overall accuracy rates of 95.78% (X-ray), 88.90% (CT scan), 83.94% (MRI), 98.89% (Ultrasound), 76.72% (OCT), 64.64% (Dermoscopy), 66.10% (Fundus), and 86.68% (Microscopy), indicating that BioMedCLIP is the most reliable model. InstructBLIP is more suited for VQA tasks that require reasoning skills, whereas OpenCLIP and ViLT perform reasonably well. From the qualitative evaluation, it is clear that the assistant can accurately identify visual findings, align image–text information, and produce clinically relevant explanations, establishing the potential of multimodal learning to support the clinical interpretation, education, and decision-making of both experts and non-experts alike.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Multimodal Intelligence for Healthcare: Combining Text and Medical Images Through Vision-Language Models

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Charanpreet Kaur,
  • Nishtha Hooda

摘要

The increasing need for the integration of different kinds of medical data, especially images and clinical documentation, is necessary for improving diagnosis, clinical decision-making, and the delivery of healthcare to patients. We are developing a Multimodal Medical AI Assistant to extract the pertinent information from a variety of medical imaging modalities using state-of-the-art Vision-Language Models such as BLIP-2, InstructBLIP, BioMedCLIP, OpenCLIP, and ViLT. We are evaluating our Multimodal Medical AI Assistant using established benchmark datasets like VQA-RAD, SLAKE, OmniMedVQA, and Medical Multimodal Evaluation Dataset. The results reveal that BioMedCLIP consistently produces the best results across all modalities with overall accuracy rates of 95.78% (X-ray), 88.90% (CT scan), 83.94% (MRI), 98.89% (Ultrasound), 76.72% (OCT), 64.64% (Dermoscopy), 66.10% (Fundus), and 86.68% (Microscopy), indicating that BioMedCLIP is the most reliable model. InstructBLIP is more suited for VQA tasks that require reasoning skills, whereas OpenCLIP and ViLT perform reasonably well. From the qualitative evaluation, it is clear that the assistant can accurately identify visual findings, align image–text information, and produce clinically relevant explanations, establishing the potential of multimodal learning to support the clinical interpretation, education, and decision-making of both experts and non-experts alike.