Visual Language Models (VLMs) are essential for various tasks, particularly the visual reasoning tasks, due to their robust multi-modal information integration, visual reasoning capabilities, and contextual awareness. However, existing VLMs’ visual spatial reasoning capabilities are often inadequate, struggling even with basic tasks such as distinguishing left from right. To address this, we propose the ZeroVLM (Code: https://github.com/zhouhao028/Iknow_up ) model, designed to enhance the visual spatial reasoning abilities of VLMs. ZeroVLM employs Zero-1-to-3, a 3D knowledge reconstruction model for obtaining different views of the input images and incorporates a view prompt to further improve visual spatial reasoning. Experimental results on four visual spatial reasoning datasets show that our ZeroVLM achieves up to 19.48% accuracy improvement, which indicates the effectiveness of 3D knowledge reconstruction and view prompt of our ZeroVLM.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

I Know About “Up”! Enhancing Spatial Reasoning in Visual Language Models Through 3D Knowledge Reconstruction

  • Hao Zhou,
  • Zaiqiao Meng,
  • Yifang Chen

摘要

Visual Language Models (VLMs) are essential for various tasks, particularly the visual reasoning tasks, due to their robust multi-modal information integration, visual reasoning capabilities, and contextual awareness. However, existing VLMs’ visual spatial reasoning capabilities are often inadequate, struggling even with basic tasks such as distinguishing left from right. To address this, we propose the ZeroVLM (Code: https://github.com/zhouhao028/Iknow_up ) model, designed to enhance the visual spatial reasoning abilities of VLMs. ZeroVLM employs Zero-1-to-3, a 3D knowledge reconstruction model for obtaining different views of the input images and incorporates a view prompt to further improve visual spatial reasoning. Experimental results on four visual spatial reasoning datasets show that our ZeroVLM achieves up to 19.48% accuracy improvement, which indicates the effectiveness of 3D knowledge reconstruction and view prompt of our ZeroVLM.