Scholarlink

<p>In recent years, prompt tuning has proven effective for adapting pre-trained vision-language models (VLMs) to downstream tasks. However, in few-shot settings, limited data often leads to overfitting and poor generalization. Additionally, the inherent differences between the textual and visual modalities highlight the need for enhanced cross-modal interactions to ensure effective integration. To address these challenges, we propose a novel approach, Parameter-Efficient Transfer Learning of Prompts and Adapters (PPA) for VLMs. PPA introduces a Prompt Interaction Module (PIM) that effectively fuses text and image features, enabling deeper cross-modal understanding. Furthermore, we design modality-specific adapters to better capture distinct representations in each branch. To preserve the general knowledge embedded in pre-trained models, we incorporate intra-modal consistency constraints during fine-tuning. Extensive experiments demonstrate that PPA outperforms state-of-the-art methods in base-to-novel generalization, cross-dataset transfer, and domain generalization, while maintaining high parameter efficiency and robustness, even in extremely data-scarce settings.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Parameter-efficient transfer learning of prompts and adapters on vision-language models

  • 期刊论文

  • 发表日期：

    2025年7月7日

查看全文
  • Ai Jian,
  • Yun Wei

摘要

In recent years, prompt tuning has proven effective for adapting pre-trained vision-language models (VLMs) to downstream tasks. However, in few-shot settings, limited data often leads to overfitting and poor generalization. Additionally, the inherent differences between the textual and visual modalities highlight the need for enhanced cross-modal interactions to ensure effective integration. To address these challenges, we propose a novel approach, Parameter-Efficient Transfer Learning of Prompts and Adapters (PPA) for VLMs. PPA introduces a Prompt Interaction Module (PIM) that effectively fuses text and image features, enabling deeper cross-modal understanding. Furthermore, we design modality-specific adapters to better capture distinct representations in each branch. To preserve the general knowledge embedded in pre-trained models, we incorporate intra-modal consistency constraints during fine-tuning. Extensive experiments demonstrate that PPA outperforms state-of-the-art methods in base-to-novel generalization, cross-dataset transfer, and domain generalization, while maintaining high parameter efficiency and robustness, even in extremely data-scarce settings.