Scholarlink

<p>Vision-Language (VL) models such as Contrastive Language-Image pretraining (CLIP) have shown remarkable zero-shot classification capabilities by jointly learning from large-scale image–text datasets using multimodal self-supervised learning (SSL). However, while these models capture strong global semantics, they often struggle with fine-grained spatial understanding, thereby limiting their effectiveness in downstream tasks like object detection and medical abnormality localization<sup><CitationRef CitationID="CR2">2</CitationRef></sup>. To address this limitation, we propose Patch-CLIP, a novel VL framework that introduces a contrastive loss aligning image patch-level embeddings with text embeddings. Unlike conventional VL approaches that only leverage global image representations, our method utilizes local patch-level features to encode spatial context, enabling effective learning of localization cues. Applied to two Chest X-ray (CXR) datasets, Patch-CLIP achieves state-of-the-art (SOTA) performance across eight abnormality detection tasks. Furthermore, the resulting patch prediction maps substantially reduce false positives at comparable sensitivity levels compared to standard saliency-based methods, providing more precise and interpretable localization of key findings. The code is available at <a href="https://github.com/Siemens-Healthineers/patch-clip">https://github.com/Siemens-Healthineers/patch-clip</a></p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

PatchCLIP enables region specific contrastive health record and image joint training with patch embedding loss

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年5月9日

查看全文
  • Sheethal Bhat,
  • Awais Mansoor,
  • Bogdan Georgescu,
  • Mathias Zinnen,
  • Pranjal Sahu,
  • Adarsh B. Panambur,
  • Florin C. Ghesu,
  • Sasa Grbic,
  • Andreas Maier

摘要

Vision-Language (VL) models such as Contrastive Language-Image pretraining (CLIP) have shown remarkable zero-shot classification capabilities by jointly learning from large-scale image–text datasets using multimodal self-supervised learning (SSL). However, while these models capture strong global semantics, they often struggle with fine-grained spatial understanding, thereby limiting their effectiveness in downstream tasks like object detection and medical abnormality localization2. To address this limitation, we propose Patch-CLIP, a novel VL framework that introduces a contrastive loss aligning image patch-level embeddings with text embeddings. Unlike conventional VL approaches that only leverage global image representations, our method utilizes local patch-level features to encode spatial context, enabling effective learning of localization cues. Applied to two Chest X-ray (CXR) datasets, Patch-CLIP achieves state-of-the-art (SOTA) performance across eight abnormality detection tasks. Furthermore, the resulting patch prediction maps substantially reduce false positives at comparable sensitivity levels compared to standard saliency-based methods, providing more precise and interpretable localization of key findings. The code is available at https://github.com/Siemens-Healthineers/patch-clip