Scholarlink

While CLIP, as a vision-language pre-trained model based on image-text pairs, exhibits powerful zero-shot transfer capabilities, its adaptation to specific applications through fine-tuning often incurs significant computational overhead. This paper proposes a new method called LoRA-CLIP, which combines text description templates generated by large language models with low-rank adaptation techniques to further enhance the performance of CLIP in zero-shot image classification tasks. Specifically, we use the LLM ‘DeepSeek’ to automatically generate text templates related to image categories, thereby enhancing the semantic expressiveness of the input text. Then, we integrate the LoRA module into the text encoder of the CLIP model, achieving efficient fine-tuning by adapting only a small number of parameters, which significantly improves the representation capability of text features. Experimental results show that LoRA-CLIP achieves outstanding performance across multiple image classification datasets. Ablation experiments further confirm the critical role of the LoRA module and DeepSeek templates in performance improvement. This method effectively reduces training costs while significantly enhancing the adaptability and practicality of the CLIP model in zero-shot image classification tasks.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

LoRA-CLIP: Low-Rank Adaptation of Text Prompts for Vision-Language Models

  • 会议论文

  • 发表日期：

    2025年

查看全文
  • Ping Zhang,
  • Haocun Li,
  • Kaiyan Song,
  • Haizheng Yu,
  • Hong Bian

摘要

While CLIP, as a vision-language pre-trained model based on image-text pairs, exhibits powerful zero-shot transfer capabilities, its adaptation to specific applications through fine-tuning often incurs significant computational overhead. This paper proposes a new method called LoRA-CLIP, which combines text description templates generated by large language models with low-rank adaptation techniques to further enhance the performance of CLIP in zero-shot image classification tasks. Specifically, we use the LLM ‘DeepSeek’ to automatically generate text templates related to image categories, thereby enhancing the semantic expressiveness of the input text. Then, we integrate the LoRA module into the text encoder of the CLIP model, achieving efficient fine-tuning by adapting only a small number of parameters, which significantly improves the representation capability of text features. Experimental results show that LoRA-CLIP achieves outstanding performance across multiple image classification datasets. Ablation experiments further confirm the critical role of the LoRA module and DeepSeek templates in performance improvement. This method effectively reduces training costs while significantly enhancing the adaptability and practicality of the CLIP model in zero-shot image classification tasks.