Background Large language models have shown potential for supporting clinical reasoning, but their performance in open-ended diagnostic tasks remains insufficiently evaluated using clinically grounded and reproducible criteria. Many prior studies have relied on multiple-choice medical benchmarks or subjective expert ratings, which may not fully reflect the uncertainty and diagnostic complexity of real-world clinical encounters. This study aimed to benchmark state-of-the-art LLMs for open-ended clinical diagnosis using a hierarchical ICD-10-CM-based evaluation framework. Methods We evaluated four LLMs: DeepSeek-V3, GPT-4o, GPT-4o mini, and Claude 3.5 Sonnet, on 50 real-world clinical cases from the publicly available MultiCaRe dataset. To reduce potential training-data leakage, only cases published after August 2024 were included. For each case, models generated three ranked differential diagnoses and corresponding ICD-10-CM codes. Diagnostic performance was assessed using a four-level hierarchical scoring framework based on ICD-10-CM structure, capturing both broad disease-category agreement and finer diagnostic specificity. Two independent runs were conducted per model-case pair to assess consistency. Descriptive statistics, pairwise one-tailed paired <InlineEquation ID="IEq1"><EquationSource Format="TEX">\(t\)</EquationSource></InlineEquation>-tests, and correlation-based analyses were used to compare performance and robustness. Results DeepSeek-V3 achieved the highest mean diagnostic score (2.32 <InlineEquation ID="IEq2"><EquationSource Format="TEX">\(\pm\)</EquationSource></InlineEquation> 1.53), followed by GPT-4o (2.23 <InlineEquation ID="IEq3"><EquationSource Format="TEX">\(\pm\)</EquationSource></InlineEquation> 1.50), Claude 3.5 Sonnet (2.12 <InlineEquation ID="IEq4"><EquationSource Format="TEX">\(\pm\)</EquationSource></InlineEquation> 1.53), and GPT-4o mini (2.10 <InlineEquation ID="IEq5"><EquationSource Format="TEX">\(\pm\)</EquationSource></InlineEquation> 1.49). Binary adjusted diagnostic accuracy was 82% for DeepSeek-V3 and GPT-4o, 80% for GPT-4o mini, and 76% for Claude 3.5 Sonnet. Across all four models, performance was generally stronger for identifying correct disease categories than for making highly specific diagnoses. DeepSeek-V3 also showed the lowest run-to-run mean difference and the highest run-to-run correlation, indicating numerically greater stability. However, overall differences between models were modest. Conclusions State-of-the-art LLMs demonstrated promising capability in open-ended clinical diagnosis, particularly for identifying clinically relevant disease categories. DeepSeek-V3 showed the strongest overall numerical performance in this benchmark, although differences across models were limited. The proposed ICD-10-CM-based hierarchical framework provides a reproducible and clinically meaningful approach for evaluating LLM diagnostic performance in open-ended settings.

DeepSeek vs ChatGPT vs Claude: benchmarking large language models for clinical diagnosis using a novel ICD-10-CM-based evaluation framework

期刊论文
出版模式：

开放获取
发表日期：

2026年5月21日

查看全文

Lichen Du,
Jiachen Zhong,
Shirong Zheng,
Liuyi Chen

摘要

Background

Large language models have shown potential for supporting clinical reasoning, but their performance in open-ended diagnostic tasks remains insufficiently evaluated using clinically grounded and reproducible criteria. Many prior studies have relied on multiple-choice medical benchmarks or subjective expert ratings, which may not fully reflect the uncertainty and diagnostic complexity of real-world clinical encounters. This study aimed to benchmark state-of-the-art LLMs for open-ended clinical diagnosis using a hierarchical ICD-10-CM-based evaluation framework.

Methods

We evaluated four LLMs: DeepSeek-V3, GPT-4o, GPT-4o mini, and Claude 3.5 Sonnet, on 50 real-world clinical cases from the publicly available MultiCaRe dataset. To reduce potential training-data leakage, only cases published after August 2024 were included. For each case, models generated three ranked differential diagnoses and corresponding ICD-10-CM codes. Diagnostic performance was assessed using a four-level hierarchical scoring framework based on ICD-10-CM structure, capturing both broad disease-category agreement and finer diagnostic specificity. Two independent runs were conducted per model-case pair to assess consistency. Descriptive statistics, pairwise one-tailed paired \(t\)-tests, and correlation-based analyses were used to compare performance and robustness.

Results

DeepSeek-V3 achieved the highest mean diagnostic score (2.32 \(\pm\) 1.53), followed by GPT-4o (2.23 \(\pm\) 1.50), Claude 3.5 Sonnet (2.12 \(\pm\) 1.53), and GPT-4o mini (2.10 \(\pm\) 1.49). Binary adjusted diagnostic accuracy was 82% for DeepSeek-V3 and GPT-4o, 80% for GPT-4o mini, and 76% for Claude 3.5 Sonnet. Across all four models, performance was generally stronger for identifying correct disease categories than for making highly specific diagnoses. DeepSeek-V3 also showed the lowest run-to-run mean difference and the highest run-to-run correlation, indicating numerically greater stability. However, overall differences between models were modest.

Conclusions

State-of-the-art LLMs demonstrated promising capability in open-ended clinical diagnosis, particularly for identifying clinically relevant disease categories. DeepSeek-V3 showed the strongest overall numerical performance in this benchmark, although differences across models were limited. The proposed ICD-10-CM-based hierarchical framework provides a reproducible and clinically meaningful approach for evaluating LLM diagnostic performance in open-ended settings.