Scholarlink

<p>Preference alignment through Direct Preference Optimization (DPO) has demonstrated significant effectiveness in aligning multimodal large language models (MLLMs) with human preferences. However, existing methods focus primarily on language preferences while neglecting the critical visual context. In this paper, we propose an <b>Ad</b>aptive <b>Vi</b>sion-enhanced <b>P</b>reference optimization (AdViP) that addresses these limitations through two key innovations: (1) vision-based preference pair construction, which integrates multiple visual foundation models to strategically remove key visual elements from the image, enhancing MLLMs’ sensitivity to visual details; and (2) adaptive preference optimization that dynamically balances vision- and language-based preferences for more accurate alignment. Extensive evaluations across different benchmarks demonstrate our effectiveness. Notably, our AdViP-7B achieves 93.7% and 96.4% reductions in response-level and mentioned-level hallucination, respectively, on the Object HalBench, significantly outperforming current state-of-the-art methods. Code is available at <a href="https://github.com/injadlu/AdViP">https://github.com/injadlu/AdViP</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

AdViP: Aligning Multi-modal LLMs via Adaptive Vision-enhanced Preference Optimization

  • 期刊论文

  • 发表日期：

    2026年4月13日

查看全文
  • Jinda Lu,
  • Jinghan Li,
  • Yuan Gao,
  • Junkang Wu,
  • Jiancan Wu,
  • Xiang Wang,
  • Xiangnan He

摘要

Preference alignment through Direct Preference Optimization (DPO) has demonstrated significant effectiveness in aligning multimodal large language models (MLLMs) with human preferences. However, existing methods focus primarily on language preferences while neglecting the critical visual context. In this paper, we propose an Adaptive Vision-enhanced Preference optimization (AdViP) that addresses these limitations through two key innovations: (1) vision-based preference pair construction, which integrates multiple visual foundation models to strategically remove key visual elements from the image, enhancing MLLMs’ sensitivity to visual details; and (2) adaptive preference optimization that dynamically balances vision- and language-based preferences for more accurate alignment. Extensive evaluations across different benchmarks demonstrate our effectiveness. Notably, our AdViP-7B achieves 93.7% and 96.4% reductions in response-level and mentioned-level hallucination, respectively, on the Object HalBench, significantly outperforming current state-of-the-art methods. Code is available at https://github.com/injadlu/AdViP.