Vision-language models (VLMs) are renowned for their extensive prior knowledge and strong generalization abilities. However, in few-shot tasks, the limited number of training samples often leads to knowledge collapse, exacerbating the model’s reliance on spurious correlations. Existing methods for few-shot classification typically do not address this issue, particularly in complex environments or with challenging data distributions. To overcome these challenges, we propose the Bias and Dynamic Prompt Contrastive Language-Image Pre-Training (BDPC) model. Our approach introduces an Attention Bias Module (ABM) to improve the model's understanding of limited samples, effectively bridging the semantic gap between images and text, and enhancing robustness to spurious correlations. Additionally, we incorporate a Dynamic Interval Prompt (DIP) to adaptively adjust the model’s behavior, mitigating the knowledge collapse issue and further improving performance. Extensive experiments across 11 diverse datasets demonstrate the effectiveness of our method.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Mitigating Spurious Correlations in Few-Shot Classification via Bias and Dynamic Prompt

  • Yalong Cheng,
  • Chuiyi Chen,
  • Zeyu Nie,
  • Zhipeng Liu,
  • Jun Liang

摘要

Vision-language models (VLMs) are renowned for their extensive prior knowledge and strong generalization abilities. However, in few-shot tasks, the limited number of training samples often leads to knowledge collapse, exacerbating the model’s reliance on spurious correlations. Existing methods for few-shot classification typically do not address this issue, particularly in complex environments or with challenging data distributions. To overcome these challenges, we propose the Bias and Dynamic Prompt Contrastive Language-Image Pre-Training (BDPC) model. Our approach introduces an Attention Bias Module (ABM) to improve the model's understanding of limited samples, effectively bridging the semantic gap between images and text, and enhancing robustness to spurious correlations. Additionally, we incorporate a Dynamic Interval Prompt (DIP) to adaptively adjust the model’s behavior, mitigating the knowledge collapse issue and further improving performance. Extensive experiments across 11 diverse datasets demonstrate the effectiveness of our method.