Contrastive Language-Image Pre-training (CLIP) represents a groundbreaking advancement in open-vocabulary zero-shot image recognitionOpen-vocabulary zero-shot image recognition, with its applications extending well beyond image-level tasks. This chapter discusses MaskCLIP, an early attempt to examine the intrinsic potential of CLIP for pixel-level dense predictionPixel-level dense prediction, specifically in semantic segmentationSemantic segmentation. With minimal modifications to CLIP, MaskCLIP shows impressive segmentation performance on open concepts across diverse datasets, all without requiring annotations or fine-tuning. In addition, through the incorporation of pseudo-labelingPseudo labeling and self-training Self-training, the enhanced MaskCLIP+ model further improves MaskCLIP under the transductive zero-shot semantic segmentation setting. We also test the robustness of MaskCLIP under input corruption and evaluate its capability in discriminating fine-grained objects and novel concepts. Our finding suggests that MaskCLIP can serve as a reliable source of supervision for dense prediction tasks to achieve annotation-free segmentation.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Unlocking CLIP for Zero-Shot Dense Segmentation

  • Chong Zhou,
  • Chen Change Loy,
  • Bo Dai

摘要

Contrastive Language-Image Pre-training (CLIP) represents a groundbreaking advancement in open-vocabulary zero-shot image recognitionOpen-vocabulary zero-shot image recognition, with its applications extending well beyond image-level tasks. This chapter discusses MaskCLIP, an early attempt to examine the intrinsic potential of CLIP for pixel-level dense predictionPixel-level dense prediction, specifically in semantic segmentationSemantic segmentation. With minimal modifications to CLIP, MaskCLIP shows impressive segmentation performance on open concepts across diverse datasets, all without requiring annotations or fine-tuning. In addition, through the incorporation of pseudo-labelingPseudo labeling and self-training Self-training, the enhanced MaskCLIP+ model further improves MaskCLIP under the transductive zero-shot semantic segmentation setting. We also test the robustness of MaskCLIP under input corruption and evaluate its capability in discriminating fine-grained objects and novel concepts. Our finding suggests that MaskCLIP can serve as a reliable source of supervision for dense prediction tasks to achieve annotation-free segmentation.