We introduce VisualQuest, a novel dataset designed to rigorously evaluate multimodal large language models (MLLMs) on abstract visual reasoning tasks that require integration of symbolic, cultural, and linguistic knowledge. Unlike existing benchmarks focused on direct image captioning or classification of realistic images, VisualQuest comprises 3,551 non-photographic, stylized images spanning four categories—Public Figures, Popular Culture, Linguistic Expressions, and Literary Works—each paired with targeted questions to probe complex reasoning. We benchmark ten state-of-the-art MLLMs, revealing that only Gemini-2.5-flash and GPT-4o achieve strong overall performance, while 3.7% of images remain unrecognized by any model, underscoring persistent challenges in multimodal understanding. Fine-grained analysis shows Gemini excels in recognizing stylized public figures, whereas GPT-4o leads in linguistic reasoning tasks such as visual puns and emoji combinations. VisualQuest thus provides a comprehensive and challenging resource for advancing research in abstract visual reasoning and highlights key areas for future model improvement. The Dataset is available at https://github.com/xkt88/VISUALQUEST .

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

VisualQuest: A Benchmark for Abstract Visual Reasoning in MLLMs

  • Kelaiti Xiao,
  • Liang Yang,
  • PAERHATI Tulajiang,
  • Dongyu Zhang,
  • Hongfei Lin

摘要

We introduce VisualQuest, a novel dataset designed to rigorously evaluate multimodal large language models (MLLMs) on abstract visual reasoning tasks that require integration of symbolic, cultural, and linguistic knowledge. Unlike existing benchmarks focused on direct image captioning or classification of realistic images, VisualQuest comprises 3,551 non-photographic, stylized images spanning four categories—Public Figures, Popular Culture, Linguistic Expressions, and Literary Works—each paired with targeted questions to probe complex reasoning. We benchmark ten state-of-the-art MLLMs, revealing that only Gemini-2.5-flash and GPT-4o achieve strong overall performance, while 3.7% of images remain unrecognized by any model, underscoring persistent challenges in multimodal understanding. Fine-grained analysis shows Gemini excels in recognizing stylized public figures, whereas GPT-4o leads in linguistic reasoning tasks such as visual puns and emoji combinations. VisualQuest thus provides a comprehensive and challenging resource for advancing research in abstract visual reasoning and highlights key areas for future model improvement. The Dataset is available at https://github.com/xkt88/VISUALQUEST .