Scholarlink

<p>Few-shot learning with vision-language models suffers from a fundamental structural limitation: support and query samples are processed through independent and asymmetric encoding pipelines. This causes query-side semantic blindness, where the model lacks rich cross-modal interactions during query encoding. Consequently, it weakens vision-language alignment and creates a training-inference distribution gap, degrading generalization to novel categories. Existing prompt-based methods inherit this asymmetry and thus cannot leverage text-conditioned semantic context on the query side at inference time. To address this limitation, we propose an Adaptive Cross-Modal Alignment via Symmetric Prompt Tuning for Few-Shot Vision–Language Learning (ACAS-PT) a unified framework that resolves this issue via symmetric prompt tuning. ACAS-PT applies identical prompt-guided, text-conditioned feature transformations to both support and query samples in a shared multimodal space, eliminating the distribution gap by design. Specifically, we propose two modules. First, a Semantic-Aware Class-Embedding Learner transforms prompt-conditioned CLIP class embeddings into class-specific semantic vectors used to modulate both support and query visual features via FiLM-based affine transformation, ensuring that query samples receive the same class-specific semantic grounding as support prototypes at inference. Second, an Adaptive Similarity Guided Module (ASGM) replaces fragile equal-weight prototype averaging with learnable instance-weighted centroid aggregation and a per-class-pair cross-modal alignment matrix that gates classification scores by within-class semantic-visual alignment confidence, yielding robust prototype estimates even under extreme label scarcity. Extensive experiments on four benchmark datasets show ACAS-PT outperforms 16 state-of-the-art methods, with symmetric processing alone yielding up to a +2.5% improvement in 5-shot accuracy. These results highlight query-side semantic blindness as a critical bottleneck in vision-language few-shot learning.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Adaptive cross-modal alignment via symmetric prompt tuning for few-shot vision–language learning

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年7月9日

查看全文
  • Feidu Akmel,
  • Xun Gong,
  • Abdalla Hadabi,
  • Abdenacer Naouri,
  • Ashraf Osman Ibrahim

摘要

Few-shot learning with vision-language models suffers from a fundamental structural limitation: support and query samples are processed through independent and asymmetric encoding pipelines. This causes query-side semantic blindness, where the model lacks rich cross-modal interactions during query encoding. Consequently, it weakens vision-language alignment and creates a training-inference distribution gap, degrading generalization to novel categories. Existing prompt-based methods inherit this asymmetry and thus cannot leverage text-conditioned semantic context on the query side at inference time. To address this limitation, we propose an Adaptive Cross-Modal Alignment via Symmetric Prompt Tuning for Few-Shot Vision–Language Learning (ACAS-PT) a unified framework that resolves this issue via symmetric prompt tuning. ACAS-PT applies identical prompt-guided, text-conditioned feature transformations to both support and query samples in a shared multimodal space, eliminating the distribution gap by design. Specifically, we propose two modules. First, a Semantic-Aware Class-Embedding Learner transforms prompt-conditioned CLIP class embeddings into class-specific semantic vectors used to modulate both support and query visual features via FiLM-based affine transformation, ensuring that query samples receive the same class-specific semantic grounding as support prototypes at inference. Second, an Adaptive Similarity Guided Module (ASGM) replaces fragile equal-weight prototype averaging with learnable instance-weighted centroid aggregation and a per-class-pair cross-modal alignment matrix that gates classification scores by within-class semantic-visual alignment confidence, yielding robust prototype estimates even under extreme label scarcity. Extensive experiments on four benchmark datasets show ACAS-PT outperforms 16 state-of-the-art methods, with symmetric processing alone yielding up to a +2.5% improvement in 5-shot accuracy. These results highlight query-side semantic blindness as a critical bottleneck in vision-language few-shot learning.