Scholarlink

<p>Vision Language Models (VLMs) with different vision encoders, trained under the two-stage paradigm of pre-training and fine-tuning, have shown varying strengths in different Visual Question Answering (VQA) tasks. However, it remains unclear whether an LLM that has already been aligned during vision-language pre-training can be reused without re-alignment when combined with a different vision encoder for a specific task. To investigate this, we systematically explore VLMs with five different vision encoders using various combinations of LLMs and pre-training strategies. Specifically, we first demonstrate that the aligned LLM with a general-purpose vision encoder can effectively enhance downstream VQA performance with task-specific encoders. Secondly, we investigate several alignment strategies between the aligned LLM and new task-specific encoders. These include <b>(i)</b> feature distillation from the projector layer using both general and task-specific encoders, <b>(ii)</b> a two-stage training strategy with varying the proportion of pre-training data. We find the aligned LLM has acquired transferable vision-language alignment capabilities, such that when combined with new encoders, it no longer requires additional alignment strategy. Evaluated across 13 task metrics after transfer learning with 5 different vision encoders, this new training recipe reduces pre-training time by 2 to 9 hours while achieving comparable or even superior performance.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

One Aligned LLM to Serve Them All: A Transfer Recipe for Training VLMs without Visual-Language Re-Alignment

  • 期刊论文

  • 发表日期：

    2026年7月1日

查看全文
  • Jiazuo Yu,
  • Yunzhi Zhuge,
  • Lu Zhang,
  • Zichen Huang,
  • Wei Zhou,
  • Dong Wang,
  • Huchuan Lu,
  • You He,
  • Long Chen

摘要

Vision Language Models (VLMs) with different vision encoders, trained under the two-stage paradigm of pre-training and fine-tuning, have shown varying strengths in different Visual Question Answering (VQA) tasks. However, it remains unclear whether an LLM that has already been aligned during vision-language pre-training can be reused without re-alignment when combined with a different vision encoder for a specific task. To investigate this, we systematically explore VLMs with five different vision encoders using various combinations of LLMs and pre-training strategies. Specifically, we first demonstrate that the aligned LLM with a general-purpose vision encoder can effectively enhance downstream VQA performance with task-specific encoders. Secondly, we investigate several alignment strategies between the aligned LLM and new task-specific encoders. These include (i) feature distillation from the projector layer using both general and task-specific encoders, (ii) a two-stage training strategy with varying the proportion of pre-training data. We find the aligned LLM has acquired transferable vision-language alignment capabilities, such that when combined with new encoders, it no longer requires additional alignment strategy. Evaluated across 13 task metrics after transfer learning with 5 different vision encoders, this new training recipe reduces pre-training time by 2 to 9 hours while achieving comparable or even superior performance.