Scholarlink

Objectives <p>To evaluate whether GPT-4.1 and Llama 3.3 70B, large language models (LLMs) assessed in zero-shot, baseline configurations, detect and categorize clinically consequential errors across types that range from pattern-based to reasoning-dependent.</p> Materials and methods <p>Two hundred fifty-six radiology reports encompassing CT (<i>n</i> = 104), MRI (<i>n</i> = 104), and X-ray (<i>n</i> = 48) studies across multiple anatomical regions were retrospectively analyzed. For each original report, four variants (<i>n</i> = 1024) were generated, each incorporating one of four predefined error types: E1, anatomical mislabeling that could cause wrong-site actions; E2, physiologically impossible or nonsensical findings; E3, diagnostic inconsistencies that affect staging or diagnosis; E4, inappropriate recommendations. The evaluated models were GPT‑4.1 04-14) and Llama 3.3 70B, both used without domain-specific training or prompt optimization to assess baseline model performance.</p> Results <p>Model performance revealed a systematic hierarchy governed by error type and imaging modality. Physiologically impossible errors (E2) showed the lowest performance: 46.2% (CT) and 33.7% (MRI) for GPT-4.1, compared with 32.7% and 25.0% for Llama 3.3, respectively. Overall success for GPT-4.1 on E2 was 16.3% (CT), 8.7% (MRI), and 12.5% (X-ray). Mislabeling errors (E1) were detected in 49.0% by GPT‑4.1 and 33.7% by Llama 3.3 for MRI. Best performance occurred for inappropriate recommendations (E4), with GPT‑4.1 achieving 85.4% detection in X-ray with high classification accuracy.</p> Conclusion <p>The evaluation framework and benchmark dataset provide a methodology for assessing LLM performance on clinically significant errors. Applied to GPT-4.1 and Llama 3.3 70B in zero-shot settings, the framework reveals a performance gap between pattern-based and reasoning-dependent error detection that warrants investigation across additional models and optimization strategies.</p> Key Points <p><Emphasis Type="BoldItalic">Question</Emphasis> <i>LLMs are increasingly used for quality assurance of radiology reports, but whether their linguistic competence translates into the detection of clinically significant errors remains unclear</i>.</p> <p><Emphasis Type="BoldItalic">Findings</Emphasis> <i>Error detection was type-dependent, both GPT-4.1 and Llama 3.3 70B performed poorly on physiologic and anatomical errors, but better on inappropriate recommendations</i>.</p> <p><Emphasis Type="BoldItalic">Clinical relevance</Emphasis> <i>LLMs failed to detect most clinically consequential errors in radiology reports, especially physiologically impossible statements that trained radiologists would rapidly identify</i>.</p> Graphical Abstract <p></p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

GPT-4.1 and Llama 3.3 70 fail to detect clinically relevant errors in radiology reports in zero-shot evaluation

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年6月19日

查看全文
  • Tugba Akinci D’Antonoli,
  • Lisa C. Adams,
  • Jannik Lübberstedt,
  • Markus M. Graf,
  • Christian J. Mertens,
  • Felix Busch,
  • Sebastian Ziegelmayer,
  • Marcus R. Makowski,
  • Keno Bressem,
  • Ina Luiken

摘要

Objectives

To evaluate whether GPT-4.1 and Llama 3.3 70B, large language models (LLMs) assessed in zero-shot, baseline configurations, detect and categorize clinically consequential errors across types that range from pattern-based to reasoning-dependent.

Materials and methods

Two hundred fifty-six radiology reports encompassing CT (n = 104), MRI (n = 104), and X-ray (n = 48) studies across multiple anatomical regions were retrospectively analyzed. For each original report, four variants (n = 1024) were generated, each incorporating one of four predefined error types: E1, anatomical mislabeling that could cause wrong-site actions; E2, physiologically impossible or nonsensical findings; E3, diagnostic inconsistencies that affect staging or diagnosis; E4, inappropriate recommendations. The evaluated models were GPT‑4.1 04-14) and Llama 3.3 70B, both used without domain-specific training or prompt optimization to assess baseline model performance.

Results

Model performance revealed a systematic hierarchy governed by error type and imaging modality. Physiologically impossible errors (E2) showed the lowest performance: 46.2% (CT) and 33.7% (MRI) for GPT-4.1, compared with 32.7% and 25.0% for Llama 3.3, respectively. Overall success for GPT-4.1 on E2 was 16.3% (CT), 8.7% (MRI), and 12.5% (X-ray). Mislabeling errors (E1) were detected in 49.0% by GPT‑4.1 and 33.7% by Llama 3.3 for MRI. Best performance occurred for inappropriate recommendations (E4), with GPT‑4.1 achieving 85.4% detection in X-ray with high classification accuracy.

Conclusion

The evaluation framework and benchmark dataset provide a methodology for assessing LLM performance on clinically significant errors. Applied to GPT-4.1 and Llama 3.3 70B in zero-shot settings, the framework reveals a performance gap between pattern-based and reasoning-dependent error detection that warrants investigation across additional models and optimization strategies.

Key Points

Question LLMs are increasingly used for quality assurance of radiology reports, but whether their linguistic competence translates into the detection of clinically significant errors remains unclear.

Findings Error detection was type-dependent, both GPT-4.1 and Llama 3.3 70B performed poorly on physiologic and anatomical errors, but better on inappropriate recommendations.

Clinical relevance LLMs failed to detect most clinically consequential errors in radiology reports, especially physiologically impossible statements that trained radiologists would rapidly identify.

Graphical Abstract