Scholarlink

<p>Open-vocabulary segmentation aims to segment images into arbitrary categories described by text, without relying on a fixed, pre-defined set of classes used in model training. More recently, with the vision-language pre-trained models such as CLIP and ALIGN demonstrating remarkable performance on zero-shot image classification tasks, some studies have adapted them to open-vocabulary semantic segmentation and achieved impressive results. However, most existing methods still require labor-intensive and time-consuming pixel-level human annotations. In this paper, we propose a novel CLIP-guided architecture with two-level semantic awareness strategy, denoted as <b>CLIP-TSA</b>, to train a segmentation network without requiring any pixel-wise annotations to achieve open-vocabulary segmentation. The two-level semantic awareness strategy comprises <b>image-level semantic awareness (ISA)</b> and <b>pixel-level semantic awareness (PSA)</b>. Its purpose is to enforce the model to learn CLIP’s powerful recognition and generalization capabilities, thereby enhancing open-vocabulary segmentation performance. Furthermore, we leverage the large-scale pre-trained Segment Anything Model (SAM) to denoise the coarse pseudo labels <b>(PLD)</b> generated by CLIP, providing more reliable supervision signals for model training. Extensive experiments demonstrate that our method outperforms previous state-of-the-art (SOTA) works in annotation-free segmentation task on the COCO Stuff and PASCAL Context datasets. Additionally, the proposed CLIP-TSA also achieves better SOTA performance than the compared models in zero-shot segmentation studies on the PASCAL VOC 2012, PASCAL Context and COCO Stuff benchmarks.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

CLIP-TSA: CLIP-guided open-vocabulary semantic segmentation with two-level semantic awareness

  • 期刊论文

  • 发表日期：

    2025年1月13日

查看全文
  • Zhixue Liang,
  • Wenyong Dong,
  • Bo Zhang

摘要

Open-vocabulary segmentation aims to segment images into arbitrary categories described by text, without relying on a fixed, pre-defined set of classes used in model training. More recently, with the vision-language pre-trained models such as CLIP and ALIGN demonstrating remarkable performance on zero-shot image classification tasks, some studies have adapted them to open-vocabulary semantic segmentation and achieved impressive results. However, most existing methods still require labor-intensive and time-consuming pixel-level human annotations. In this paper, we propose a novel CLIP-guided architecture with two-level semantic awareness strategy, denoted as CLIP-TSA, to train a segmentation network without requiring any pixel-wise annotations to achieve open-vocabulary segmentation. The two-level semantic awareness strategy comprises image-level semantic awareness (ISA) and pixel-level semantic awareness (PSA). Its purpose is to enforce the model to learn CLIP’s powerful recognition and generalization capabilities, thereby enhancing open-vocabulary segmentation performance. Furthermore, we leverage the large-scale pre-trained Segment Anything Model (SAM) to denoise the coarse pseudo labels (PLD) generated by CLIP, providing more reliable supervision signals for model training. Extensive experiments demonstrate that our method outperforms previous state-of-the-art (SOTA) works in annotation-free segmentation task on the COCO Stuff and PASCAL Context datasets. Additionally, the proposed CLIP-TSA also achieves better SOTA performance than the compared models in zero-shot segmentation studies on the PASCAL VOC 2012, PASCAL Context and COCO Stuff benchmarks.