Scholarlink

<p>Medical image classification plays a vital role in healthcare by aiding diagnosis and treatment planning. Vision-language models, such as CLIP, demonstrate promising capabilities in medical imaging by enabling zero-shot learning, which mitigates the need for extensive labeled data. Test-Time Prompt Tuning (TPT) improves CLIP by dynamically refining prompts during inference to enhance performance. Yet, TPT’s reliance on entropy minimization can exacerbate overconfidence, impairing the model’s calibration, and leading to unreliable predictions. However, traditional calibration methods rely on substantial amounts of labeled data, making them impractical for test-time scenarios. In this work, we propose Information-Theoretic-Based Test-Time Prompt Tuning (I-TPT) to address these calibration challenges. We introduce a novel Mutual Dissimilarity Loss (MDL) inspired by information theory to enhance the separation between prompt embeddings, ensuring minimal overlap between classes. By maximizing dissimilarity, I-TPT reduces overconfidence and improves calibration, ensuring safer and more reliable clinical decision-making. Extensive experiments on multiple medical datasets demonstrate that our approach significantly lowers the Expected Calibration Error (ECE) while maintaining high classification accuracy. I-TPT offers a scalable solution for deploying vision-language models in medical applications, providing both robustness and adaptability without the computational burden of full retraining.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Calibrated test-time prompt tuning for vision-language models on medicine via information theoretic

  • 期刊论文

  • 发表日期：

    2026年2月28日

查看全文
  • Chenyu Ge,
  • Yitong Li,
  • Guangle Song

摘要

Medical image classification plays a vital role in healthcare by aiding diagnosis and treatment planning. Vision-language models, such as CLIP, demonstrate promising capabilities in medical imaging by enabling zero-shot learning, which mitigates the need for extensive labeled data. Test-Time Prompt Tuning (TPT) improves CLIP by dynamically refining prompts during inference to enhance performance. Yet, TPT’s reliance on entropy minimization can exacerbate overconfidence, impairing the model’s calibration, and leading to unreliable predictions. However, traditional calibration methods rely on substantial amounts of labeled data, making them impractical for test-time scenarios. In this work, we propose Information-Theoretic-Based Test-Time Prompt Tuning (I-TPT) to address these calibration challenges. We introduce a novel Mutual Dissimilarity Loss (MDL) inspired by information theory to enhance the separation between prompt embeddings, ensuring minimal overlap between classes. By maximizing dissimilarity, I-TPT reduces overconfidence and improves calibration, ensuring safer and more reliable clinical decision-making. Extensive experiments on multiple medical datasets demonstrate that our approach significantly lowers the Expected Calibration Error (ECE) while maintaining high classification accuracy. I-TPT offers a scalable solution for deploying vision-language models in medical applications, providing both robustness and adaptability without the computational burden of full retraining.