Scholarlink

Purpose <p>A comprehensive performance evaluation of seven state-of-the-art large language models (LLMs) compared to human resident benchmarks on ophthalmology board exam questions. Furthermore, this study aims to define current model strengths and limitations while validating an assessment tool for clinicians to evaluate model-generated outputs.</p> Methods <p>Seven LLMs were assessed: ChatGPT-5, ChatGPT-4, Gemini 2.5 Pro, Gemini 2.5 Flash, Claude Sonnet 4.5, Grok-4-Fast-Reasoning, and Perplexity Sonar Pro. A dataset of 1,037 Israeli ophthalmology board questions (2020–2025) was manually categorized by question type (logical vs. informative), image modality, and 12 subspecialties. Models were evaluated for accuracy compared with resident performance, response latency, question difficulty, and self-assessed confidence.</p> Results <p>Performance varied substantially across models. Gemini 2.5 Pro achieved the highest accuracy, followed by ChatGPT-5, both outperforming residents. Accuracy declined significantly with increasing question difficulty (Very Hard vs. Easy: aOR 0.28, 95% CI,0.16–0.46; aP &lt; 0.001), mirroring resident trends. Questions containing images were significantly more challenging for all models. Logical reasoning questions were significantly more challenging for LLMs than informative ones (aOR 0.66, aP = 0.019). Confidence calibration varied widely, with ChatGPT-5 showing superior discrimination (AUC = 0.827).</p> Conclusions <p>LLMs’ performance on ophthalmology board questions varied across models. Gemini 2.5 Pro achieved the highest performance, surpassing the mean resident score. However, the majority of LLMs exhibit significant error rates and poor calibration of self-assessed confidence. To safely integrate LLMs into the clinical workflow, continuous evaluation is essential to guide ophthalmologists on LLMs’ characteristics and clinical limitations.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

A large comprehensive comparison of large language models on ophthalmology board exams

  • 期刊论文

  • 发表日期：

    2026年5月14日

查看全文
  • Alon Moore Galindo,
  • Razan Saadi,
  • Tomer Kerman,
  • David Schwartzman,
  • Doron Pasternak,
  • Michael Kinori,
  • Anat Loewenstein

摘要

Purpose

A comprehensive performance evaluation of seven state-of-the-art large language models (LLMs) compared to human resident benchmarks on ophthalmology board exam questions. Furthermore, this study aims to define current model strengths and limitations while validating an assessment tool for clinicians to evaluate model-generated outputs.

Methods

Seven LLMs were assessed: ChatGPT-5, ChatGPT-4, Gemini 2.5 Pro, Gemini 2.5 Flash, Claude Sonnet 4.5, Grok-4-Fast-Reasoning, and Perplexity Sonar Pro. A dataset of 1,037 Israeli ophthalmology board questions (2020–2025) was manually categorized by question type (logical vs. informative), image modality, and 12 subspecialties. Models were evaluated for accuracy compared with resident performance, response latency, question difficulty, and self-assessed confidence.

Results

Performance varied substantially across models. Gemini 2.5 Pro achieved the highest accuracy, followed by ChatGPT-5, both outperforming residents. Accuracy declined significantly with increasing question difficulty (Very Hard vs. Easy: aOR 0.28, 95% CI,0.16–0.46; aP < 0.001), mirroring resident trends. Questions containing images were significantly more challenging for all models. Logical reasoning questions were significantly more challenging for LLMs than informative ones (aOR 0.66, aP = 0.019). Confidence calibration varied widely, with ChatGPT-5 showing superior discrimination (AUC = 0.827).

Conclusions

LLMs’ performance on ophthalmology board questions varied across models. Gemini 2.5 Pro achieved the highest performance, surpassing the mean resident score. However, the majority of LLMs exhibit significant error rates and poor calibration of self-assessed confidence. To safely integrate LLMs into the clinical workflow, continuous evaluation is essential to guide ophthalmologists on LLMs’ characteristics and clinical limitations.