Scholarlink

Background <p>Large language models (LLM) have shown promise in assisting medical decision-making. However, there is limited literature exploring the diagnostic accuracy of LLMs in generating differential diagnoses from text-based image descriptions and clinical presentations in pediatric radiology.</p> Objective <p>To examine the performance of multiple proprietary LLMs in producing accurate differential diagnoses for text-based pediatric radiological cases without imaging.</p> Materials and methods <p>One hundred sixty-four cases were retrospectively selected from a pediatric radiology textbook and converted into two formats: (1) image description only, and (2) image description with clinical presentation. The ChatGPT-4&#xa0;V, Claude 3.5 Sonnet, and Gemini 1.5 Pro algorithms were given these inputs and tasked with providing a top 1 diagnosis and a top 3 differential diagnoses. Accuracy of responses was assessed by comparison with the original literature. Top 1 accuracy was defined as whether the top 1 diagnosis matched the textbook, and top 3 differential accuracy was defined as the number of diagnoses in the model-generated top 3 differential that matched any of the top 3 diagnoses in the textbook. McNemar’s test, Cochran’s <i>Q</i> test, Friedman test, and Wilcoxon signed-rank test were used to compare algorithms and assess the impact of added clinical information, respectively.</p> Results <p>There was no significant difference in top 1 accuracy between ChatGPT-4&#xa0;V, Claude 3.5 Sonnet, and Gemini 1.5 Pro when only image descriptions were provided (56.1% [95% CI 48.4–63.5], 64.6% [95% CI 57.1–71.5], 61.6% [95% CI 54.0–68.7]; <i>P</i> = 0.11). Adding clinical presentation to image description significantly improved top 1 accuracy for ChatGPT-4&#xa0;V (64.0% [95% CI 56.4–71.0], <i>P</i> = 0.02) and Claude 3.5 Sonnet (80.5% [95% CI 73.8–85.8], <i>P</i> &lt; 0.001). For image description and clinical presentation cases, Claude 3.5 Sonnet significantly outperformed both ChatGPT-4&#xa0;V and Gemini 1.5 Pro (<i>P</i> &lt; 0.001). For top 3 differential accuracy, no significant differences were observed between ChatGPT-4&#xa0;V, Claude 3.5 Sonnet, and Gemini 1.5 Pro, regardless of whether the cases included only image descriptions (1.29 [95% CI 1.16–1.41], 1.35 [95% CI 1.23–1.48], 1.37 [95% CI 1.25–1.49]; <i>P</i> = 0.60) or both image descriptions and clinical presentations (1.33 [95% CI 1.20–1.45], 1.52 [95% CI 1.41–1.64], 1.48 [95% 1.36–1.59]; <i>P</i> = 0.72). Only Claude 3.5 Sonnet performed significantly better when clinical presentation was added (<i>P</i> &lt; 0.001).</p> Conclusion <p>Commercial LLMs performed similarly on pediatric radiology cases in providing top 1 accuracy and top 3 differential accuracy when only a text-based image description was used. Adding clinical presentation significantly improved top 1 accuracy for ChatGPT-4&#xa0;V and Claude 3.5 Sonnet, with Claude showing the largest improvement. Claude 3.5 Sonnet outperformed both ChatGPT-4&#xa0;V and Gemini 1.5 Pro in top 1 accuracy when both image and clinical data were provided. No significant differences were found in top 3 differential accuracy across models in any condition.</p> Graphical Abstract <p></p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Accuracy of large language models in generating differential diagnosis from clinical presentation and imaging findings in pediatric cases

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年7月12日

查看全文
  • Jinho Jung,
  • Michael Phillipi,
  • Bryant Tran,
  • Kasha Chen,
  • Nathan Chan,
  • Erwin Ho,
  • Shawn Sun,
  • Roozbeh Houshyar

摘要

Background

Large language models (LLM) have shown promise in assisting medical decision-making. However, there is limited literature exploring the diagnostic accuracy of LLMs in generating differential diagnoses from text-based image descriptions and clinical presentations in pediatric radiology.

Objective

To examine the performance of multiple proprietary LLMs in producing accurate differential diagnoses for text-based pediatric radiological cases without imaging.

Materials and methods

One hundred sixty-four cases were retrospectively selected from a pediatric radiology textbook and converted into two formats: (1) image description only, and (2) image description with clinical presentation. The ChatGPT-4 V, Claude 3.5 Sonnet, and Gemini 1.5 Pro algorithms were given these inputs and tasked with providing a top 1 diagnosis and a top 3 differential diagnoses. Accuracy of responses was assessed by comparison with the original literature. Top 1 accuracy was defined as whether the top 1 diagnosis matched the textbook, and top 3 differential accuracy was defined as the number of diagnoses in the model-generated top 3 differential that matched any of the top 3 diagnoses in the textbook. McNemar’s test, Cochran’s Q test, Friedman test, and Wilcoxon signed-rank test were used to compare algorithms and assess the impact of added clinical information, respectively.

Results

There was no significant difference in top 1 accuracy between ChatGPT-4 V, Claude 3.5 Sonnet, and Gemini 1.5 Pro when only image descriptions were provided (56.1% [95% CI 48.4–63.5], 64.6% [95% CI 57.1–71.5], 61.6% [95% CI 54.0–68.7]; P = 0.11). Adding clinical presentation to image description significantly improved top 1 accuracy for ChatGPT-4 V (64.0% [95% CI 56.4–71.0], P = 0.02) and Claude 3.5 Sonnet (80.5% [95% CI 73.8–85.8], P < 0.001). For image description and clinical presentation cases, Claude 3.5 Sonnet significantly outperformed both ChatGPT-4 V and Gemini 1.5 Pro (P < 0.001). For top 3 differential accuracy, no significant differences were observed between ChatGPT-4 V, Claude 3.5 Sonnet, and Gemini 1.5 Pro, regardless of whether the cases included only image descriptions (1.29 [95% CI 1.16–1.41], 1.35 [95% CI 1.23–1.48], 1.37 [95% CI 1.25–1.49]; P = 0.60) or both image descriptions and clinical presentations (1.33 [95% CI 1.20–1.45], 1.52 [95% CI 1.41–1.64], 1.48 [95% 1.36–1.59]; P = 0.72). Only Claude 3.5 Sonnet performed significantly better when clinical presentation was added (P < 0.001).

Conclusion

Commercial LLMs performed similarly on pediatric radiology cases in providing top 1 accuracy and top 3 differential accuracy when only a text-based image description was used. Adding clinical presentation significantly improved top 1 accuracy for ChatGPT-4 V and Claude 3.5 Sonnet, with Claude showing the largest improvement. Claude 3.5 Sonnet outperformed both ChatGPT-4 V and Gemini 1.5 Pro in top 1 accuracy when both image and clinical data were provided. No significant differences were found in top 3 differential accuracy across models in any condition.

Graphical Abstract