Scholarlink

Multimodal Large Language Models (MLLMs) represent cutting-edge AI technology, yet fundamental vulnerabilities in vision encoding persist across diverse architectures. We investigate these core limitations through systematic analysis of contemporary open source models, specifically DeepSeek Janus Pro and VL2, selected for their architectural diversity and state-of-the-art performance. Despite their different design approaches, specifically decoupled visual encoding versus dynamic tiling with MoE computation, both models exhibit a critical vulnerability: a disconnect between visual similarity and representational similarity in vision encoders. Through adapted embedding manipulation attacks, we demonstrate that visually similar images produce drastically different patch embeddings, while semantically distant concepts occupy unexpectedly proximate regions in embedding space. Our geometric analysis reveals non linear optimization trajectories and dual coding mechanisms through magnitude redistribution, where high magnitude components preserve structural fidelity while amplified low magnitude components inject target semantics. Across COCO, DALL \(\cdot \) E 3, and SVIT datasets, we achieve hallucination rates up to 98.0% while maintaining high visual fidelity (SSIM > 0.88), with our novel LLaMA 3.1 based multi prompt detection framework revealing consistently higher vulnerability in closed form versus open ended questioning. These findings demonstrate that architectural improvements alone cannot address core vision encoding limitations, emphasizing the need for fundamental advances in vision language representation alignment to secure MLLM deployment.

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Vision Embeddings and Their Role in Hallucination Vulnerabilities of Multimodal Large Language Models

  • 会议论文

  • 发表日期：

    2026年

查看全文
  • Chashi Mahiul Islam,
  • Samuel Jacob Chacko,
  • Preston Horne,
  • Xiuwen Liu

摘要

Multimodal Large Language Models (MLLMs) represent cutting-edge AI technology, yet fundamental vulnerabilities in vision encoding persist across diverse architectures. We investigate these core limitations through systematic analysis of contemporary open source models, specifically DeepSeek Janus Pro and VL2, selected for their architectural diversity and state-of-the-art performance. Despite their different design approaches, specifically decoupled visual encoding versus dynamic tiling with MoE computation, both models exhibit a critical vulnerability: a disconnect between visual similarity and representational similarity in vision encoders. Through adapted embedding manipulation attacks, we demonstrate that visually similar images produce drastically different patch embeddings, while semantically distant concepts occupy unexpectedly proximate regions in embedding space. Our geometric analysis reveals non linear optimization trajectories and dual coding mechanisms through magnitude redistribution, where high magnitude components preserve structural fidelity while amplified low magnitude components inject target semantics. Across COCO, DALL \(\cdot \) E 3, and SVIT datasets, we achieve hallucination rates up to 98.0% while maintaining high visual fidelity (SSIM > 0.88), with our novel LLaMA 3.1 based multi prompt detection framework revealing consistently higher vulnerability in closed form versus open ended questioning. These findings demonstrate that architectural improvements alone cannot address core vision encoding limitations, emphasizing the need for fundamental advances in vision language representation alignment to secure MLLM deployment.