Large vision-language models (LVLMs) have achieved impressive performance on tasks such as visual question answering and image captioning, yet they still suffer from hallucinations, producing descriptions that conflict with the visual input. Prior efforts to mitigate hallucinations often rely on data-centric strategies or specialized fine-tuning, which require large-scale annotations or costly retraining. More recent training-free methods adjust attention at the modality level but largely ignore fine-grained object–region grounding. In this work, we analyze hallucinations through the lens of cross-modal attention and show that, during decoding, LVLMs tend to collapse onto a few visually global tokens, while progressive encoding compresses visual evidence into global tokens that are easily overridden by language priors. To address this, we propose a training-free framework that (i) leverages CLIP patch embeddings and early-layer attention maps to reweight decoder cross-attention in an object-centric, fine-grained manner, and (ii) introduces an auxiliary decoding branch with masked global tokens for contrastive decoding, effectively reducing hallucinations without additional training.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Fine-Grained Attention Enhancement for Mitigating Hallucinations in LVLMs

  • Jidong Yang,
  • Hongxun Yao,
  • Xi Chen,
  • Shouxu Jiang

摘要

Large vision-language models (LVLMs) have achieved impressive performance on tasks such as visual question answering and image captioning, yet they still suffer from hallucinations, producing descriptions that conflict with the visual input. Prior efforts to mitigate hallucinations often rely on data-centric strategies or specialized fine-tuning, which require large-scale annotations or costly retraining. More recent training-free methods adjust attention at the modality level but largely ignore fine-grained object–region grounding. In this work, we analyze hallucinations through the lens of cross-modal attention and show that, during decoding, LVLMs tend to collapse onto a few visually global tokens, while progressive encoding compresses visual evidence into global tokens that are easily overridden by language priors. To address this, we propose a training-free framework that (i) leverages CLIP patch embeddings and early-layer attention maps to reweight decoder cross-attention in an object-centric, fine-grained manner, and (ii) introduces an auxiliary decoding branch with masked global tokens for contrastive decoding, effectively reducing hallucinations without additional training.