Scholarlink

Purpose <p>To compare the accuracy and reproducibility of lesion-diameter measurements performed by three state-of-the-art LLMs with those obtained by radiologists.</p> Materials and methods <p>In this retrospective study using a public database, 83 patients with solitary colorectal-cancer liver metastases were identified. From each CT series, a radiologist extracted the single axial slice showing the maximal tumor diameter and converted it to a 512 × 512-pixel PNG image (window level 50 HU, window width 400 HU) with pixel size encoded in the filename. Three LLMs—ChatGPT-o3 (OpenAI), Gemini 2.5 Pro (Google), and Claude 4 Opus (Anthropic)—were prompted to estimate the longest lesion diameter twice, ≥ 1&#xa0;week apart. Two board-certified radiologists (12&#xa0;years’ experience each) independently measured the same single slice images and one radiologist repeated the measurements after ≥ 1&#xa0;week. Agreement was assessed with intraclass correlation coefficients (ICC); 95% confidence intervals were obtained by bootstrap resampling (5 000 iterations).</p> Results <p>Radiologist inter-observer agreement was excellent (ICC = 0.95, 95% CI 0.86–0.99); intra-observer agreement was 0.98 (95% CI 0.94–0.99). Gemini achieved good model-to-radiologist agreement (ICC = 0.81, 95% CI 0.68–0.89) and intra-model reproducibility (ICC = 0.78, 95% CI 0.65–0.87). GPT-o3 showed moderate agreement (ICC = 0.52) and poor reproducibility (ICC = 0.25); Claude showed poor agreement (ICC = 0.07) and reproducibility (ICC = 0.47).</p> Conclusion <p>LLMs do not yet match radiologists in measuring colorectal cancer liver metastasis; however, Gemini’s good agreement and reproducibility highlight the rapid progress of image interpretation capability of LLMs.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Accuracy and reproducibility of large language model measurements of liver metastases: comparison with radiologist measurements

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2025年10月4日

查看全文
  • Haruto Sugawara,
  • Akiyo Takada,
  • Shimpei Kato

摘要

Purpose

To compare the accuracy and reproducibility of lesion-diameter measurements performed by three state-of-the-art LLMs with those obtained by radiologists.

Materials and methods

In this retrospective study using a public database, 83 patients with solitary colorectal-cancer liver metastases were identified. From each CT series, a radiologist extracted the single axial slice showing the maximal tumor diameter and converted it to a 512 × 512-pixel PNG image (window level 50 HU, window width 400 HU) with pixel size encoded in the filename. Three LLMs—ChatGPT-o3 (OpenAI), Gemini 2.5 Pro (Google), and Claude 4 Opus (Anthropic)—were prompted to estimate the longest lesion diameter twice, ≥ 1 week apart. Two board-certified radiologists (12 years’ experience each) independently measured the same single slice images and one radiologist repeated the measurements after ≥ 1 week. Agreement was assessed with intraclass correlation coefficients (ICC); 95% confidence intervals were obtained by bootstrap resampling (5 000 iterations).

Results

Radiologist inter-observer agreement was excellent (ICC = 0.95, 95% CI 0.86–0.99); intra-observer agreement was 0.98 (95% CI 0.94–0.99). Gemini achieved good model-to-radiologist agreement (ICC = 0.81, 95% CI 0.68–0.89) and intra-model reproducibility (ICC = 0.78, 95% CI 0.65–0.87). GPT-o3 showed moderate agreement (ICC = 0.52) and poor reproducibility (ICC = 0.25); Claude showed poor agreement (ICC = 0.07) and reproducibility (ICC = 0.47).

Conclusion

LLMs do not yet match radiologists in measuring colorectal cancer liver metastasis; however, Gemini’s good agreement and reproducibility highlight the rapid progress of image interpretation capability of LLMs.