Scholarlink

<p>Parameter-Efficient Tuning (PET) has been widely studied to adapt pretrained vision-language models (VLMs) to various downstream tasks. Most existing PET methods are limited to uni-modal tuning of textual modality and cannot fully exploit the generalization ability of VLMs. Recent multi-modal tuning methods cannot address the catastrophic performance loss in generalizing to unseen classes without incurring significant memory and computation overheads. In this paper, we reveal the multi-modal forgetting issue that violates model coupling for constraining visual features and textual embeddings in simultaneously tuning both modalities, and propose a novel <b>Vi</b>si<b>o</b>n <b>L</b>anguage <b>E</b>fficient <b>T</b>uning (<b>VioLET</b>) framework to address the issue. We first propose VioLET-CMG that leverages <b>C</b>ollaborative <b>M</b>ulti-modal <b>G</b>radients (<b>CMG</b>) by incorporating an additional visual encoder to provide extra textual gradients to regulate the multi-modal tuning. VioLET-CMG avoids conflicts between visual and textual gradients and guarantee modal coupling with textual gradient orthogonalization. Furthermore, we present VioLET-CarE that achieves quasi-orthogonal textual tuning by <b>C</b>ontrolling l<b>a</b>nguage hype<b>r</b>spherical <b>E</b>nergy (<b>CarE</b>) to remove the additional visual encoder for enabling collaborative multi-modal gradients in VioLET-CMG. VioLET-CarE enhances computational efficiency via additive low-rank feature transformation for textual tuning. Experimental results demonstrate that the proposed VioLET framework consistently achieves state-of-the-art performance in new class generalization and few-shot recognition using both ResNet-50 and ViT/B-16 backbones. Remarkably, compared with VioLET-CMG, VioLET-CarE further reduces GPU memory cost by 27% and training times by 36% on average with enhanced generalization ability.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Vision-Language Efficient Tuning for Mitigating Catastrophic Forgetting in Multi-Modal Learning

  • 期刊论文

  • 发表日期：

    2026年3月9日

查看全文
  • Yaoming Wang,
  • Yuchen Liu,
  • Wenrui Dai,
  • Xiaopeng Zhang,
  • Han Li,
  • Chenglin Li,
  • Junni Zou,
  • Qi Tian,
  • Hongkai Xiong

摘要

Parameter-Efficient Tuning (PET) has been widely studied to adapt pretrained vision-language models (VLMs) to various downstream tasks. Most existing PET methods are limited to uni-modal tuning of textual modality and cannot fully exploit the generalization ability of VLMs. Recent multi-modal tuning methods cannot address the catastrophic performance loss in generalizing to unseen classes without incurring significant memory and computation overheads. In this paper, we reveal the multi-modal forgetting issue that violates model coupling for constraining visual features and textual embeddings in simultaneously tuning both modalities, and propose a novel Vision Language Efficient Tuning (VioLET) framework to address the issue. We first propose VioLET-CMG that leverages Collaborative Multi-modal Gradients (CMG) by incorporating an additional visual encoder to provide extra textual gradients to regulate the multi-modal tuning. VioLET-CMG avoids conflicts between visual and textual gradients and guarantee modal coupling with textual gradient orthogonalization. Furthermore, we present VioLET-CarE that achieves quasi-orthogonal textual tuning by Controlling language hyperspherical Energy (CarE) to remove the additional visual encoder for enabling collaborative multi-modal gradients in VioLET-CMG. VioLET-CarE enhances computational efficiency via additive low-rank feature transformation for textual tuning. Experimental results demonstrate that the proposed VioLET framework consistently achieves state-of-the-art performance in new class generalization and few-shot recognition using both ResNet-50 and ViT/B-16 backbones. Remarkably, compared with VioLET-CMG, VioLET-CarE further reduces GPU memory cost by 27% and training times by 36% on average with enhanced generalization ability.