Scholarlink

<p>Current evaluations of Large Language Models (LLMs) in medicine primarily focus on text-based benchmarks, leaving their multimodal diagnostic capabilities in complex, real-world clinical scenarios largely undefined. Furthermore, comparisons against large-scale human benchmarks remain scarce. To address this gap, we conducted a comprehensive evaluation of state-of-the-art multimodal LLMs (GPT-4o, Claude 3.7, and Doubao) using 272 complex cases from the New England Journal of Medicine Image Challenge (2009–2025). Uniquely, we benchmarked AI performance against a massive global dataset of 16,401,888 physician responses, representing the largest comparative study of human-AI diagnostic reasoning to date. Strikingly, all multimodal LLMs significantly outperformed the global physician collective (P &lt; 0.001). Claude 3.7 achieved a diagnostic accuracy of 89.0%, surpassing the physician majority vote (46.7%) by an absolute margin of over 40 percentage points. Even in challenging cases where human accuracy fell below 40%, Claude 3.7 maintained an accuracy of 86.5%. A novel finding of this study is the remarkably low concordance between high-performing models and physicians (Cohen’s κ: 0.08–0.24). The ratio of model-advantage to physician-advantage cases reached 15.4:1, suggesting that MLLMs succeed in distinct areas where human cognition often falters. Our findings demonstrate that MLLMs have reached a superhuman tier in multimodal diagnostic accuracy. The substantial performance gap, coupled with low human-AI concordance, implies that MLLMs do not merely replicate human knowledge but utilize fundamentally distinct and complementary diagnostic reasoning pathways. These results position multimodal LLMs as critical, independent second readers capable of augmenting clinical decision-making in diagnostically difficult scenarios.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Multimodal large language models challenge NEJM image challenge

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年2月10日

查看全文
  • Chiyu Sheng,
  • Shumin Shen,
  • Lin Wang,
  • Jie Chen,
  • Wei Chen,
  • Shanghu Wang,
  • Nianfei Wang

摘要

Current evaluations of Large Language Models (LLMs) in medicine primarily focus on text-based benchmarks, leaving their multimodal diagnostic capabilities in complex, real-world clinical scenarios largely undefined. Furthermore, comparisons against large-scale human benchmarks remain scarce. To address this gap, we conducted a comprehensive evaluation of state-of-the-art multimodal LLMs (GPT-4o, Claude 3.7, and Doubao) using 272 complex cases from the New England Journal of Medicine Image Challenge (2009–2025). Uniquely, we benchmarked AI performance against a massive global dataset of 16,401,888 physician responses, representing the largest comparative study of human-AI diagnostic reasoning to date. Strikingly, all multimodal LLMs significantly outperformed the global physician collective (P < 0.001). Claude 3.7 achieved a diagnostic accuracy of 89.0%, surpassing the physician majority vote (46.7%) by an absolute margin of over 40 percentage points. Even in challenging cases where human accuracy fell below 40%, Claude 3.7 maintained an accuracy of 86.5%. A novel finding of this study is the remarkably low concordance between high-performing models and physicians (Cohen’s κ: 0.08–0.24). The ratio of model-advantage to physician-advantage cases reached 15.4:1, suggesting that MLLMs succeed in distinct areas where human cognition often falters. Our findings demonstrate that MLLMs have reached a superhuman tier in multimodal diagnostic accuracy. The substantial performance gap, coupled with low human-AI concordance, implies that MLLMs do not merely replicate human knowledge but utilize fundamentally distinct and complementary diagnostic reasoning pathways. These results position multimodal LLMs as critical, independent second readers capable of augmenting clinical decision-making in diagnostically difficult scenarios.