Contrastive Language-Image Pretraining (CLIP) has demonstrated superior generalization capabilities in image classification over general visual domains. However, for some specific downstream tasks, CLIP still suffers from unsatisfactory results due to semantic gaps. To this end, CoOp has been proposed to adopt learnable prompt tokens for few-shot learning. In this chapter, we showcase an alternative to utilize feature adapters to enhance the few-shot image classification performance of CLIP. We first introduce CLIP-Adapter, an adapter architecture equipped after CLIP’s vision encoder for injecting downstream knowledge. On top of that, we further present two upgraded variants, Tip-Adapter and Tip-Adapter-F, a training-free adaption method, and its fine-tuning variant. They incorporate the few-shot semantics with the pretrained CLIP with a constructed cache model, which further achieve both effectiveness and efficiency for leading image classification performance.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Learning Efficient Feature Adapters for Vision-Language Models

  • Renrui Zhang,
  • Peng Gao

摘要

Contrastive Language-Image Pretraining (CLIP) has demonstrated superior generalization capabilities in image classification over general visual domains. However, for some specific downstream tasks, CLIP still suffers from unsatisfactory results due to semantic gaps. To this end, CoOp has been proposed to adopt learnable prompt tokens for few-shot learning. In this chapter, we showcase an alternative to utilize feature adapters to enhance the few-shot image classification performance of CLIP. We first introduce CLIP-Adapter, an adapter architecture equipped after CLIP’s vision encoder for injecting downstream knowledge. On top of that, we further present two upgraded variants, Tip-Adapter and Tip-Adapter-F, a training-free adaption method, and its fine-tuning variant. They incorporate the few-shot semantics with the pretrained CLIP with a constructed cache model, which further achieve both effectiveness and efficiency for leading image classification performance.