SCP: Sinkhorn-reconciled Collaborative Prompt Learning for Vision-Language Models
摘要
Prompt learning and Adapter applied to vision-language models are increasingly becoming a focus of research. We propose Sinkhorn-reconciled Collaborative Prompt Learning (SCP), a novel approach for fine-tuning vision-language models. Our approach aims to improve the generalization ability of the foundational vision-language models when fine-tuning downstream visual tasks in a few-shot setting. We introduce attention variants Correlative Self-Attention (CSA) and Merge Attention (MA) to design CorrAttentionMapper (CAM) and MergeBoostAdapter (MBAdapter) modules for the prompt and adapter, respectively, to enhance the prompt features expression capability and synergy between the two modalities. In addition, we reconcile the general and prompted features through Sinkhorn divergence to constrain the consistency of the model output distribution and prevent overfitting in the downstream task. Our method is fully evaluated on three benchmark tasks, including base-to-novel generalization, cross-dataset evaluation and domain generalization. Experimental results are compared with state-of-the-art prompt learning methods to demonstrate the effectiveness of the method.