Can multi-modal large language models (MLLMs) that can “see” an image be said to “understand” it? This work proposes that MLLMs may be trapped in a Visual Room, capable of mechanically processing and describing every detail of visual inputs without genuinely comprehending their underlying intentions. This Visual Room Argument (VRA) challenges the prevailing assumption that perceptual proficiency equates to true understanding, highlighting a potential perception–cognition gap in current MLLMs. To examine this hypothesis, we introduce a two-tier evaluation framework spanning perception and cognition. The perception component evaluates whether MLLMs can accurately capture the surface-level details of visual contents, where the cognitive component examines their ability to infer sarcasm polarity. To support this framework, we construct a high-quality multi-modal sarcasm dataset consisting of 924 static images and 100 dynamic videos. All sarcasm labels are annotated by the original authors and verified by independent reviewers to ensure clarity and consistency. We evaluate eight state-of-the-art (SoTA) MLLMs on this benchmark. While these models demonstrate strong performance in visual perception, they exhibit a high error rate ( \(\sim \) 17.1%) in sarcasm understanding, revealing a significant gap between seeing and understanding. These findings provide empirical grounding for the proposed VRA (Dataset and codes are available at https://github.com/annomary/MMSar ).

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Are MLLMs Trapped in the Visual Room?

  • Yazhou Zhang,
  • Chunwang Zou,
  • Qimeng Liu,
  • Lu Rong,
  • Ben Yao,
  • Zheng Lian,
  • Qiuchi Li,
  • Peng Zhang,
  • Jing Qin

摘要

Can multi-modal large language models (MLLMs) that can “see” an image be said to “understand” it? This work proposes that MLLMs may be trapped in a Visual Room, capable of mechanically processing and describing every detail of visual inputs without genuinely comprehending their underlying intentions. This Visual Room Argument (VRA) challenges the prevailing assumption that perceptual proficiency equates to true understanding, highlighting a potential perception–cognition gap in current MLLMs. To examine this hypothesis, we introduce a two-tier evaluation framework spanning perception and cognition. The perception component evaluates whether MLLMs can accurately capture the surface-level details of visual contents, where the cognitive component examines their ability to infer sarcasm polarity. To support this framework, we construct a high-quality multi-modal sarcasm dataset consisting of 924 static images and 100 dynamic videos. All sarcasm labels are annotated by the original authors and verified by independent reviewers to ensure clarity and consistency. We evaluate eight state-of-the-art (SoTA) MLLMs on this benchmark. While these models demonstrate strong performance in visual perception, they exhibit a high error rate ( \(\sim \) 17.1%) in sarcasm understanding, revealing a significant gap between seeing and understanding. These findings provide empirical grounding for the proposed VRA (Dataset and codes are available at https://github.com/annomary/MMSar ).