Scholarlink

<p>Prompt learning has become crucial for adapting Visual Language Models (VLM) to downstream tasks. Although existing prompt learning models have made significant strides, they still face two major challenges: 1. Too much attention is paid to learning about basic classes, making it harder to understand novel classes; 2. Most methods only rely on the context information provided by the prompt template, resulting in limited text features. In this study, we propose a new fine-tuning method for Visual-Language Models called Input-Enhanced Prompt Tuning (IEPT). The IEPT improves the generalization of VLMs for downstream tasks by introducing two components, i.e., the Data Augmentation Framework (DAF) and the Category Generalization Optimizer (CGO). Specifically, the DAF employs Large Language Models to resolve issues of word ambiguity by obtaining more class label context, and uses simple image augmentation to address the issue of limited features by providing more image samples. The CGO prevents overfitting by adding new class names during training. Experiments show that the performance of IEPT in various evaluation suites is better or comparable to that of the existing method, covering basic to novel generalization, domain generalization, and cross-dataset evaluation. Compared to the state-of-the-art method PromptSRC, IEPT achieves an absolute improvement of 0.40% for base classes, 1.56% for novel classes and 1.04% on the harmonic mean, averaged over 11 datasets. In addition, we present detailed ablation studies that validate the individual contributions of DAF and CGO to the overall performance of IEPT. Our code is available at <a href="https://github.com/ayuan0626/IEPT">https://github.com/ayuan0626/IEPT</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

IEPT: input-enhanced prompt tuning for visual-language models

  • 期刊论文

  • 发表日期：

    2025年10月17日

查看全文
  • Chunru Dong,
  • Junyuan Liu,
  • Qiang Hua,
  • Jiahong Tang,
  • Feng Zhang

摘要

Prompt learning has become crucial for adapting Visual Language Models (VLM) to downstream tasks. Although existing prompt learning models have made significant strides, they still face two major challenges: 1. Too much attention is paid to learning about basic classes, making it harder to understand novel classes; 2. Most methods only rely on the context information provided by the prompt template, resulting in limited text features. In this study, we propose a new fine-tuning method for Visual-Language Models called Input-Enhanced Prompt Tuning (IEPT). The IEPT improves the generalization of VLMs for downstream tasks by introducing two components, i.e., the Data Augmentation Framework (DAF) and the Category Generalization Optimizer (CGO). Specifically, the DAF employs Large Language Models to resolve issues of word ambiguity by obtaining more class label context, and uses simple image augmentation to address the issue of limited features by providing more image samples. The CGO prevents overfitting by adding new class names during training. Experiments show that the performance of IEPT in various evaluation suites is better or comparable to that of the existing method, covering basic to novel generalization, domain generalization, and cross-dataset evaluation. Compared to the state-of-the-art method PromptSRC, IEPT achieves an absolute improvement of 0.40% for base classes, 1.56% for novel classes and 1.04% on the harmonic mean, averaged over 11 datasets. In addition, we present detailed ablation studies that validate the individual contributions of DAF and CGO to the overall performance of IEPT. Our code is available at https://github.com/ayuan0626/IEPT.