Scholarlink

<p>This study evaluated the performance of three large language models, including ChatGPT-4o, ChatGPT-5, and Gemini 2.5 Flash, on 532 Persian multiple-choice questions from the 2025 Iranian surgical subspecialty board examinations. Questions spanned five domains: Pediatric, Cardiovascular, Vascular and Endovascular, Thoracic, and Plastic &amp; Reconstructive Surgery. Using standardized prompts, we assessed overall accuracy, variation across subspecialties and question types, and the effect of question length. Gemini 2.5 Flash and ChatGPT-5 achieved higher accuracy (73.9% and 73.3%) than ChatGPT-4o (68.2%). Agreement with the official key was substantial for Gemini 2.5 Flash (κ = 0.651) and ChatGPT-5 (κ = 0.642), and moderate to substantial for ChatGPT-4o (κ = 0.575). Model performance was stable across subspecialties, but all three showed lower accuracy on surgical technique questions compared with clinical scenarios or basic science items. Question length did not affect ChatGPT-5 or Gemini 2.5 Flash, while longer stems reduced ChatGPT-4o’s performance. These findings indicate that newer LLMs provide measurable improvements in surgical question answering, though lower performance on surgical technique items supports cautious integration and further multimodal development.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Benchmarking large language models on persian surgical subspecialty board examinations: a comparative study of ChatGPT-4o, ChatGPT-5, and Gemini 2.5 Flash

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年5月8日

查看全文
  • Shahab Sheikhalishahi,
  • Farzad Rafiei,
  • Seyed Masoud Hosseini,
  • Alireza Haddadi,
  • Saina Sadeghipour

摘要

This study evaluated the performance of three large language models, including ChatGPT-4o, ChatGPT-5, and Gemini 2.5 Flash, on 532 Persian multiple-choice questions from the 2025 Iranian surgical subspecialty board examinations. Questions spanned five domains: Pediatric, Cardiovascular, Vascular and Endovascular, Thoracic, and Plastic & Reconstructive Surgery. Using standardized prompts, we assessed overall accuracy, variation across subspecialties and question types, and the effect of question length. Gemini 2.5 Flash and ChatGPT-5 achieved higher accuracy (73.9% and 73.3%) than ChatGPT-4o (68.2%). Agreement with the official key was substantial for Gemini 2.5 Flash (κ = 0.651) and ChatGPT-5 (κ = 0.642), and moderate to substantial for ChatGPT-4o (κ = 0.575). Model performance was stable across subspecialties, but all three showed lower accuracy on surgical technique questions compared with clinical scenarios or basic science items. Question length did not affect ChatGPT-5 or Gemini 2.5 Flash, while longer stems reduced ChatGPT-4o’s performance. These findings indicate that newer LLMs provide measurable improvements in surgical question answering, though lower performance on surgical technique items supports cautious integration and further multimodal development.