Scholarlink

Purpose <p>Large language models (LLMs) are increasingly utilized in medicine; however, their reliability in surgical and diagnostic specialties requiring guideline-anchored reasoning remains underexplored. The study systematically evaluated the performance of five contemporary LLMs in otorhinolaryngology (ENT).</p> Methods <p>A prospective, two-phase, cross-sectional analysis compared ChatGPT-5 (OpenAI), Gemini-3 (Google), Grok 2 (xAI), Llama 4 (Meta), and DeepSeek-4 (DeepSeek) using 100 standardized ENT clinical scenarios derived from AAO-HNS, ERS, and IFOS guidelines. Each model answered 900 questions (4 open-ended and 5 multiple-choice questions per case), and the process was repeated one month later to assess temporal stability. Two board-certified otorhinolaryngologists, blinded to the study, rated all 9,000 responses for accuracy, comprehensiveness, and adherence to guidelines on a 6-point scale. Statistical analyses included Kruskal–Wallis, Dunn’s post-hoc, Wilcoxon, and ICC reliability testing.</p> Results <p>Inter-rater reliability was excellent (ICC = 0.91; Cronbach’s α = 0.89). Significant inter-model differences were found across all domains (<i>p</i> &lt; 0.001). ChatGPT-5 achieved the highest composite score (5.58 ± 0.24), outperforming Gemini-3 (5.42 ± 0.28; <i>p</i> &lt; 0.01) and other models within a narrow, clinically credible band (5.05–5.30). Accuracy was higher for multiple-choice than open-ended items (<i>p</i> = 0.027), with ChatGPT-5 showing the smallest format gap. All models demonstrated strong alignment with international ENT guidelines and adhered to clinical safety standards. Temporal stability was greatest in ChatGPT-5 and Gemini-3, with minimal drift between test phases (Δ &lt; 0.1).</p> Conclusion <p>All evaluated LLMs delivered high factual and guideline-based reasoning in otolaryngology, with ChatGPT-5 exhibiting superior precision, completeness, and longitudinal stability. These findings support the supervised integration of LLMs as adjunctive tools for clinical education and decision support in ENT.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Benchmarking next-generation large language models in otorhinolaryngology: a prospective, guideline-anchored, blinded evaluation of ChatGPT-5, Gemini-3, Grok-2, Llama-4, and DeepSeek-4 across case scenarios

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年4月29日

查看全文
  • Behçet Günsoy,
  • Koray Tümüklü

摘要

Purpose

Large language models (LLMs) are increasingly utilized in medicine; however, their reliability in surgical and diagnostic specialties requiring guideline-anchored reasoning remains underexplored. The study systematically evaluated the performance of five contemporary LLMs in otorhinolaryngology (ENT).

Methods

A prospective, two-phase, cross-sectional analysis compared ChatGPT-5 (OpenAI), Gemini-3 (Google), Grok 2 (xAI), Llama 4 (Meta), and DeepSeek-4 (DeepSeek) using 100 standardized ENT clinical scenarios derived from AAO-HNS, ERS, and IFOS guidelines. Each model answered 900 questions (4 open-ended and 5 multiple-choice questions per case), and the process was repeated one month later to assess temporal stability. Two board-certified otorhinolaryngologists, blinded to the study, rated all 9,000 responses for accuracy, comprehensiveness, and adherence to guidelines on a 6-point scale. Statistical analyses included Kruskal–Wallis, Dunn’s post-hoc, Wilcoxon, and ICC reliability testing.

Results

Inter-rater reliability was excellent (ICC = 0.91; Cronbach’s α = 0.89). Significant inter-model differences were found across all domains (p < 0.001). ChatGPT-5 achieved the highest composite score (5.58 ± 0.24), outperforming Gemini-3 (5.42 ± 0.28; p < 0.01) and other models within a narrow, clinically credible band (5.05–5.30). Accuracy was higher for multiple-choice than open-ended items (p = 0.027), with ChatGPT-5 showing the smallest format gap. All models demonstrated strong alignment with international ENT guidelines and adhered to clinical safety standards. Temporal stability was greatest in ChatGPT-5 and Gemini-3, with minimal drift between test phases (Δ < 0.1).

Conclusion

All evaluated LLMs delivered high factual and guideline-based reasoning in otolaryngology, with ChatGPT-5 exhibiting superior precision, completeness, and longitudinal stability. These findings support the supervised integration of LLMs as adjunctive tools for clinical education and decision support in ENT.