Scholarlink

In this paper we present how vision language models can be specialized for medical image interpretation using supervised fine-tuning. Access to reliable expertise is limited by high costs or long waiting times and general purpose language models lack domain-specific knowledge. We address these problems by fine-tuning Llama 3.2 Vision 11B Instruct on the MEDPIX-ClinQA dataset, which contains 20,500 question-answer pairs across various imaging modalities. To enable efficient learning, we employ a curriculum approach, where we allow the model to understand factual medical knowledge and terminology and then we introduce it to complex clinical analysis. For computational efficiency, we use Parameter-Efficient Fine-Tuning (PEFT) techniques, particularly QLoRA. We evaluate visual reasoning capabilities using BERTScore, which compares the semantic similarity between generated answers and ground truth. We see an improvements of 49.8% on precision, 26.3% in recall and 38.6% in F1 score. This suggests multimodal models have the potential to be specialized for medical tasks using domain-specific fine-tuning and help physicians in providing better diagnoses.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Fine-Tuning Vision Language Models for Medical Visual Question Answering

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Victor Teslaru,
  • Gabriel Pojoga,
  • Ștefan-Daniel Achirei

摘要

In this paper we present how vision language models can be specialized for medical image interpretation using supervised fine-tuning. Access to reliable expertise is limited by high costs or long waiting times and general purpose language models lack domain-specific knowledge. We address these problems by fine-tuning Llama 3.2 Vision 11B Instruct on the MEDPIX-ClinQA dataset, which contains 20,500 question-answer pairs across various imaging modalities. To enable efficient learning, we employ a curriculum approach, where we allow the model to understand factual medical knowledge and terminology and then we introduce it to complex clinical analysis. For computational efficiency, we use Parameter-Efficient Fine-Tuning (PEFT) techniques, particularly QLoRA. We evaluate visual reasoning capabilities using BERTScore, which compares the semantic similarity between generated answers and ground truth. We see an improvements of 49.8% on precision, 26.3% in recall and 38.6% in F1 score. This suggests multimodal models have the potential to be specialized for medical tasks using domain-specific fine-tuning and help physicians in providing better diagnoses.