Scholarlink

The scarcity of labeled medical images presents challenges for machine learning applications in medical diagnosis. Annotating medical image datasets requires domain-specific expertise, making the process time-consuming, legally sensitive, and expensive. However, while medical images are often unlabeled, they frequently have corresponding textual diagnoses. In this study, we use Contrastive Language-Image Pretraining (CLIP) to improve both the classification and clustering of medical images. By training a CLIP-like model on the ROCOv2 dataset with ResNet50 and DistilBERT encoders, we enhance classification performance, achieving a significant improvement over the general-purpose OpenAI CLIP. Additionally, we employ CLIP-based embeddings for unsupervised clustering using K-means and compare the results with traditional K-means applied to raw image features. Our approach demonstrates superior cluster coherence and alignment with diagnostic categories, highlighting the effectiveness of multi-modal embeddings in structuring medical data. These findings underscore the potential of CLIP models in advancing both supervised and unsupervised medical image analysis.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Multi-modal Contrastive Learning for Medical Image Classification and Clustering

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Zinaid Kapić,
  • Ivan Štajduhar

摘要

The scarcity of labeled medical images presents challenges for machine learning applications in medical diagnosis. Annotating medical image datasets requires domain-specific expertise, making the process time-consuming, legally sensitive, and expensive. However, while medical images are often unlabeled, they frequently have corresponding textual diagnoses. In this study, we use Contrastive Language-Image Pretraining (CLIP) to improve both the classification and clustering of medical images. By training a CLIP-like model on the ROCOv2 dataset with ResNet50 and DistilBERT encoders, we enhance classification performance, achieving a significant improvement over the general-purpose OpenAI CLIP. Additionally, we employ CLIP-based embeddings for unsupervised clustering using K-means and compare the results with traditional K-means applied to raw image features. Our approach demonstrates superior cluster coherence and alignment with diagnostic categories, highlighting the effectiveness of multi-modal embeddings in structuring medical data. These findings underscore the potential of CLIP models in advancing both supervised and unsupervised medical image analysis.