Enhancing Continual Learning in Vision-Language Models Using Shared Adapter Integrated with Contrastive Learning
摘要
This paper presents an approach to enhance continual learning in vision-language models through shared adapters and contrastive learning. Our method leverages a frozen, pre-trained Vision Transformer (ViT) model, while only updating the trainable adapters. To address catastrophic forgetting, we utilize online prototypes—a lightweight representation for storing class information instead of maintaining a traditional memory bank of previous task data. The shared adapter architecture is integrated with contrastive learning, which helps improve model plasticity and reinforces class boundaries without requiring access to previously learned data. Experimental results on ImageNet-A, ImageNet-R, and CIFAR100 datasets demonstrate that our approach achieves competitive performance compared to some state-of-the-art methods on the last score, with accuracy improvements of up to 62.74% on ImageNet-A and 91.93% on CIFAR100 for the previous task, while maintaining a competitive average performance across all tasks.