Scholarlink

<p>The rapid development of multimodal large language models (MLLMs) raises the question of how they compare to human performance. While existing datasets often feature synthetic or overly simplistic tasks, some models have already surpassed human expert baselines. In this paper, we present MULTI, a Chinese multimodal dataset derived from authentic examination questions. Comprising over 18000 carefully selected and refined questions, MULTI evaluates models using real-world examination standards, encompassing image-text comprehension, complex reasoning, and knowledge recall. Additionally, we also introduce MULTI-Elite, a 500-question selected hard subset, and MULTI-Extend with more than 4500 external knowledge context pieces for testing in-context learning capabilities. Our evaluation highlights substantial room for MLLM advancement, with Qwen2-VL-72B achieving a 76.9% accuracy on MULTI and 53.1% on MULTI-Elite leading 25 evaluated models, compared to human expert baselines of 86.1% and 73.1%. MULTI serves not only as a robust evaluation platform but also paves the way for the development of expert-level AI. Details and access are available at <a href="https://OpenDFM.github.io/MULTI-Benchmark">https://OpenDFM.github.io/MULTI-Benchmark</a>.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

MULTI: multimodal understanding leaderboard with text and images

  • 期刊论文

  • 发表日期：

    2025年9月28日

查看全文
  • Zichen Zhu,
  • Yang Xu,
  • Lu Chen,
  • Jingkai Yang,
  • Yichuan Ma,
  • Yiming Sun,
  • Hailin Wen,
  • Jiaqi Liu,
  • Jinyu Cai,
  • Yingzi Ma,
  • Situo Zhang,
  • Zihan Zhao,
  • Liangtai Sun,
  • Kai Yu

摘要

The rapid development of multimodal large language models (MLLMs) raises the question of how they compare to human performance. While existing datasets often feature synthetic or overly simplistic tasks, some models have already surpassed human expert baselines. In this paper, we present MULTI, a Chinese multimodal dataset derived from authentic examination questions. Comprising over 18000 carefully selected and refined questions, MULTI evaluates models using real-world examination standards, encompassing image-text comprehension, complex reasoning, and knowledge recall. Additionally, we also introduce MULTI-Elite, a 500-question selected hard subset, and MULTI-Extend with more than 4500 external knowledge context pieces for testing in-context learning capabilities. Our evaluation highlights substantial room for MLLM advancement, with Qwen2-VL-72B achieving a 76.9% accuracy on MULTI and 53.1% on MULTI-Elite leading 25 evaluated models, compared to human expert baselines of 86.1% and 73.1%. MULTI serves not only as a robust evaluation platform but also paves the way for the development of expert-level AI. Details and access are available at https://OpenDFM.github.io/MULTI-Benchmark.