Scholarlink

Background <p>Diagnostic errors and delays are common in rheumatology, driven by overlapping symptoms and the rarity of many diseases. While traditional diagnostic decision support systems (DDSS) have seen limited adoption because of high input burden and low perceived value, large language models (LLMs) now offer genuine dialogue and reduced effort, with rapidly improving diagnostic performance, yet empirical evidence on their real-world effectiveness and educational impact is still scarce.</p> Objective <p>The aim of this study was to investigate the impact of an LLM on medical students’ diagnostic performance in rheumatology compared with traditional resources.</p> Methods <p>In this randomized controlled trial, medical students solved three rheumatology vignettes. For each case, they provided a main diagnosis with confidence and up to four differential diagnoses. Participants were randomized to use ChatGPT-4o plus traditional resources or traditional resources alone. The primary outcome was the proportion of correct top diagnoses. Secondary outcomes were correctness within the top 5 diagnoses, a cumulative diagnostic score, diagnostic confidence, and completion time.</p> Results <p>Sixty-eight students (mean [SD] age 24.8 [2.6] years) were randomized. The LLM group identified the correct top diagnosis more often than controls (77.5% vs. 32.4%), yielding an adjusted odds ratio of 7.0 (95% CI 3.8–14.4; <i>P</i>&lt;.001), and also exceeded LLM-only performance (77.5% vs. 71.6%). Cumulative diagnostic scores were higher with LLM support (mean [SD] 12.3 [2.3] vs. 6.7 [3.2]; <i>P</i>&lt;.001), as was confidence (7.0 [1.3] vs. 6.1 [1.2]; <i>P</i>&lt;.001). Completion time increased in the LLM group (505 [131] s vs. 287 [106] s; <i>P</i>&lt;.001).</p> Conclusions <p>Medical students using an LLM achieved significantly higher diagnostic accuracy than those using conventional resources. Students assisted by the LLM also outperformed the model alone, highlighting the potential of human-AI collaboration. These findings suggest that LLMs may help improve clinical reasoning in complex fields such as rheumatology. However, these findings should be interpreted cautiously, as larger and more diverse studies are needed to confirm their generalisability.</p> Trial registration <p>ClinicalTrials.gov, NCT06748170 registered 27 December 2024.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Large language models enhance diagnostic reasoning of medical students in rheumatology: a randomized controlled trial

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年3月25日

查看全文
  • Anna Roemer,
  • Nadine Schlicker,
  • Anna Kernder,
  • Benedikt Albe,
  • Juliana Hack,
  • Martin Hirsch,
  • Andreas Mayr,
  • Sebastian Kuhn,
  • Johannes Knitza

摘要

Background

Diagnostic errors and delays are common in rheumatology, driven by overlapping symptoms and the rarity of many diseases. While traditional diagnostic decision support systems (DDSS) have seen limited adoption because of high input burden and low perceived value, large language models (LLMs) now offer genuine dialogue and reduced effort, with rapidly improving diagnostic performance, yet empirical evidence on their real-world effectiveness and educational impact is still scarce.

Objective

The aim of this study was to investigate the impact of an LLM on medical students’ diagnostic performance in rheumatology compared with traditional resources.

Methods

In this randomized controlled trial, medical students solved three rheumatology vignettes. For each case, they provided a main diagnosis with confidence and up to four differential diagnoses. Participants were randomized to use ChatGPT-4o plus traditional resources or traditional resources alone. The primary outcome was the proportion of correct top diagnoses. Secondary outcomes were correctness within the top 5 diagnoses, a cumulative diagnostic score, diagnostic confidence, and completion time.

Results

Sixty-eight students (mean [SD] age 24.8 [2.6] years) were randomized. The LLM group identified the correct top diagnosis more often than controls (77.5% vs. 32.4%), yielding an adjusted odds ratio of 7.0 (95% CI 3.8–14.4; P<.001), and also exceeded LLM-only performance (77.5% vs. 71.6%). Cumulative diagnostic scores were higher with LLM support (mean [SD] 12.3 [2.3] vs. 6.7 [3.2]; P<.001), as was confidence (7.0 [1.3] vs. 6.1 [1.2]; P<.001). Completion time increased in the LLM group (505 [131] s vs. 287 [106] s; P<.001).

Conclusions

Medical students using an LLM achieved significantly higher diagnostic accuracy than those using conventional resources. Students assisted by the LLM also outperformed the model alone, highlighting the potential of human-AI collaboration. These findings suggest that LLMs may help improve clinical reasoning in complex fields such as rheumatology. However, these findings should be interpreted cautiously, as larger and more diverse studies are needed to confirm their generalisability.

Trial registration

ClinicalTrials.gov, NCT06748170 registered 27 December 2024.