Scholarlink

Objective <p>To evaluate the performance of two large language models (LLMs) in extracting and assigning the American College of Radiology Thyroid Imaging Reporting and Data System (ACR TI-RADS) categories from free-text thyroid ultrasound (US) reports.</p> Methods <p>This retrospective study included thyroid US reports from patients who underwent thyroid examinations. Three experienced US radiologists (with &gt; 15 years of experience) and three general radiologists (with 4–5 years of experience) reviewed these free-text reports and assigned an ACR TI-RADS category based on the original US descriptions. The same reports were processed using standardized prompts in GPT-4.1 and Gemini-2.5-pro. Inter-rater agreement was evaluated using Gwet’s AC1 coefficient.</p> Results <p>Inter-rater agreement between human readers was moderate (Gwet’s AC1 = 0.64, 95% CI 0.62–0.68) and was significantly higher than that between two LLMs (Gwet’s AC1 = 0.47, 95% CI 0.44–0.50; <i>P</i> &lt; 0.001). Agreement between expert radiologists and the LLMs was substantial (Gwet’s AC1 = 0.67, 95% CI 0.65–0.70 for GPT-4.1 and 0.66, 95% CI 0.63–0.69 for Gemini-2.5-pro), whereas agreement with general readers was lower (Gwet’s AC1 = 0.49, 95% CI 0.46–0.52 for GPT-4.1, and 0.47, 95% CI 0.44–0.50 for Gemini-2.5-pro). The AUCs were 0.859 (95% CI, 0.831–0.887) for expert radiologists, 0.827 (95% CI, 0.798–0.856) for general radiologists, 0.836 (95% CI, 0.813–0.859) for Gemini-2.5-pro, and 0.811 (95% CI, 0.782–0.839) for GPT-4.1.</p> Conclusion <p>LLMs demonstrated promising performance in assigning TI-RADS category from free-text thyroid US reports, achieving accuracy and agreement comparable to that of general radiologists. However, they are not yet ready to replace expert interpretation in high-stakes clinical settings.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Evaluation of large language models for TI-RADS category extraction from free-text thyroid US reports: a comparative study with human readers

  • 期刊论文

  • 发表日期：

    2026年5月2日

查看全文
  • Anding Dong,
  • Lanlan Zhang,
  • Jian Ma,
  • Wei Li,
  • Guohui Hong

摘要

Objective

To evaluate the performance of two large language models (LLMs) in extracting and assigning the American College of Radiology Thyroid Imaging Reporting and Data System (ACR TI-RADS) categories from free-text thyroid ultrasound (US) reports.

Methods

This retrospective study included thyroid US reports from patients who underwent thyroid examinations. Three experienced US radiologists (with > 15 years of experience) and three general radiologists (with 4–5 years of experience) reviewed these free-text reports and assigned an ACR TI-RADS category based on the original US descriptions. The same reports were processed using standardized prompts in GPT-4.1 and Gemini-2.5-pro. Inter-rater agreement was evaluated using Gwet’s AC1 coefficient.

Results

Inter-rater agreement between human readers was moderate (Gwet’s AC1 = 0.64, 95% CI 0.62–0.68) and was significantly higher than that between two LLMs (Gwet’s AC1 = 0.47, 95% CI 0.44–0.50; P < 0.001). Agreement between expert radiologists and the LLMs was substantial (Gwet’s AC1 = 0.67, 95% CI 0.65–0.70 for GPT-4.1 and 0.66, 95% CI 0.63–0.69 for Gemini-2.5-pro), whereas agreement with general readers was lower (Gwet’s AC1 = 0.49, 95% CI 0.46–0.52 for GPT-4.1, and 0.47, 95% CI 0.44–0.50 for Gemini-2.5-pro). The AUCs were 0.859 (95% CI, 0.831–0.887) for expert radiologists, 0.827 (95% CI, 0.798–0.856) for general radiologists, 0.836 (95% CI, 0.813–0.859) for Gemini-2.5-pro, and 0.811 (95% CI, 0.782–0.839) for GPT-4.1.

Conclusion

LLMs demonstrated promising performance in assigning TI-RADS category from free-text thyroid US reports, achieving accuracy and agreement comparable to that of general radiologists. However, they are not yet ready to replace expert interpretation in high-stakes clinical settings.