Vision-language models (VLMs) have shown remarkable progress by leveraging large-scale pre-training on multimodal data. However, fully fine-tuning these massive models for downstream tasks is computationally expensive. This survey comprehensively reviews the latest advancements in prompt tuning for VLMs. We categorize prompt tuning into steady prompts using fixed prompts and dynamic prompts with learnable tokens. Various prompt utilization strategies across different vision-language tasks like visual recognition, question answering, referring expression comprehension, and zero-shot learning are analyzed in-depth. Key innovations include probabilistic and task residual tuning, read-only optimization for few-shot learning, in-context prompt learning, and multitask tuning. We benchmark performance across standard datasets, identifying gaps between few-shot and fine-tuning. Challenges include data and computation limitations. By unifying perspectives on this rapidly evolving field, this survey provides a timely synthesis and roadmap for future research into efficient, robust, and versatile prompt tuning for vision-language understanding.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

A Comprehensive Review of Prompt Tuning in Vision-Language Models

  • Lovee,
  • Harsh Verma,
  • Aruna Malik

摘要

Vision-language models (VLMs) have shown remarkable progress by leveraging large-scale pre-training on multimodal data. However, fully fine-tuning these massive models for downstream tasks is computationally expensive. This survey comprehensively reviews the latest advancements in prompt tuning for VLMs. We categorize prompt tuning into steady prompts using fixed prompts and dynamic prompts with learnable tokens. Various prompt utilization strategies across different vision-language tasks like visual recognition, question answering, referring expression comprehension, and zero-shot learning are analyzed in-depth. Key innovations include probabilistic and task residual tuning, read-only optimization for few-shot learning, in-context prompt learning, and multitask tuning. We benchmark performance across standard datasets, identifying gaps between few-shot and fine-tuning. Challenges include data and computation limitations. By unifying perspectives on this rapidly evolving field, this survey provides a timely synthesis and roadmap for future research into efficient, robust, and versatile prompt tuning for vision-language understanding.