Scholarlink

<p>Vision–language models (VLMs) pre-trained on large-scale image–text pairs have shown impressive results in zero-shot vision tasks. Knowledge transferability of these models can be further improved with the help of a limited number of samples. Feature adapter tuning is a prominent approach employed for efficient transfer learning (ETL). However, most of the previous ETL models focus on tuning either prior-independent or prior-dependent feature adapters. We propose a novel ETL approach that leverages both adapter styles simultaneously. Additionally, most existing ETL models rely on using textual prompts constructed by completing general pre-defined templates. This approach neglects the descriptive knowledge that can assist VLM by presenting an informative prompt. Instead of pre-defined templates for prompt construction, we use a pre-trained LLM to generate attribute-specific prompts for each visual category. Furthermore, we guide the VLM with context-aware discriminative information generated by the pre-trained LLM to emphasize features that distinguish the most probable candidate classes. The proposed ETL model is evaluated on 11 datasets and sets a new state of the art. Our code and all collected prompts are publicly available at <a href="https://github.com/mrzarei5/DATViL">https://github.com/mrzarei5/DATViL</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Dual Adapter Tuning of Vision–Language Models Using Large Language Models

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年5月8日

查看全文
  • Mohammad Reza Zarei,
  • Abbas Akkasi,
  • Majid Komeili

摘要

Vision–language models (VLMs) pre-trained on large-scale image–text pairs have shown impressive results in zero-shot vision tasks. Knowledge transferability of these models can be further improved with the help of a limited number of samples. Feature adapter tuning is a prominent approach employed for efficient transfer learning (ETL). However, most of the previous ETL models focus on tuning either prior-independent or prior-dependent feature adapters. We propose a novel ETL approach that leverages both adapter styles simultaneously. Additionally, most existing ETL models rely on using textual prompts constructed by completing general pre-defined templates. This approach neglects the descriptive knowledge that can assist VLM by presenting an informative prompt. Instead of pre-defined templates for prompt construction, we use a pre-trained LLM to generate attribute-specific prompts for each visual category. Furthermore, we guide the VLM with context-aware discriminative information generated by the pre-trained LLM to emphasize features that distinguish the most probable candidate classes. The proposed ETL model is evaluated on 11 datasets and sets a new state of the art. Our code and all collected prompts are publicly available at https://github.com/mrzarei5/DATViL.