<p>Long-tailed recognition is a very challenging research problem with a wide range of applications in the real world. Although CLIP-based methods have achieved state-of-the-art recognition accuracy, they are often quite complex and incur high computational costs. This raises a key question: how can we reduce complexity while maintaining competitive recognition accuracy for long-tailed recognition? To address this problem, we propose a new framework, Multi-Prompt Long-Tailed Recognition (MPLR), which utilizes a pre-trained vision-language model for long-tailed recognition. MPLR consists of two components: Multimodal Prompt Tuning (MPT) and Dynamic Visual Prompt Generator (DVPG). MPT fine-tunes both the visual and textual branches of a pre-trained model by introducing visual and textual prompts. DVPG focuses on identifying tail classes with limited samples by integrating image prompt features and task-related textual prompt features obtained from MPT. Compared with some state-of-the-art methods, our framework has two distinctive characteristics: (1) The parameter size of the model is small and the efficiency is very high. (2) The model has very strong competitive recognition accuracy. The extensive experiments demonstrate that MPLR significantly enhances tail class accuracy with improved training efficiency. Our code is available at <a href="https://github.com/SuperLuckyQiu/MPLR">https://github.com/SuperLuckyQiu/MPLR</a>.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

MPLR: a long-tailed recognition method based on visual language prompts

  • Shaoxin Qiu,
  • Junhai Zhai,
  • Jiankai Chen

摘要

Long-tailed recognition is a very challenging research problem with a wide range of applications in the real world. Although CLIP-based methods have achieved state-of-the-art recognition accuracy, they are often quite complex and incur high computational costs. This raises a key question: how can we reduce complexity while maintaining competitive recognition accuracy for long-tailed recognition? To address this problem, we propose a new framework, Multi-Prompt Long-Tailed Recognition (MPLR), which utilizes a pre-trained vision-language model for long-tailed recognition. MPLR consists of two components: Multimodal Prompt Tuning (MPT) and Dynamic Visual Prompt Generator (DVPG). MPT fine-tunes both the visual and textual branches of a pre-trained model by introducing visual and textual prompts. DVPG focuses on identifying tail classes with limited samples by integrating image prompt features and task-related textual prompt features obtained from MPT. Compared with some state-of-the-art methods, our framework has two distinctive characteristics: (1) The parameter size of the model is small and the efficiency is very high. (2) The model has very strong competitive recognition accuracy. The extensive experiments demonstrate that MPLR significantly enhances tail class accuracy with improved training efficiency. Our code is available at https://github.com/SuperLuckyQiu/MPLR.