Objective To compare open-weight and proprietary large language models (LLMs),a rule-based extractor (RBE) and radiologists for labelling pulmonary embolismCT reports, and to test whether a hybrid RBE–LLM workflow improveslabelling performance. Materials and methods This single-centre retrospective study included structured CTreports from October 2021 to March 2025. Three labelling pipelines wereevaluated: an RBE; a model-agnostic LLM extractor (18 open-weight, four GPT-4variants); and a hybrid pipeline routing only RBE failures to an LLM. Groundtruth was defined at the report-text level by deterministic schema matching forinitially RBE-valid fields and blinded adjudication of RBE-invalid fields by twoattending radiologists. Eight radiologists provided a human baseline. Outcomesincluded F1 scores, accuracy, LLM-based salvage of RBE failures, and labellingtime. Results In total, 2,923 reports from 2,923 patients (mean age 66 ± 17 years;1,465 women) were included. Falcon3-10b and GPT-4.1-mini achieved similaritem-level performance (F1 0.98 [95% CI, 0.97–0.98] for both; p = 0.70) and both exceeded the RBE (F1 0.81 [95%CI, 0.80–0.82]; p < 0.001).Salvage of RBE failures was comparable between open-weight and proprietarymodels (88.1% vs 91.9%; p = 0.12). The hybrid RBE–LLM workflowachieved 99.8% accuracy and F1 0.99 (0.98–0.99), exceeding both the RBEand pooled radiologists (F1 0.92 [95% CI, 0.90–0.93]; all p < 0.001). Conclusion Schema-constrained open-weight and proprietary LLMs exceededrule-based extraction and, at the upper end of performance, matched a pooledradiologist label-transfer baseline. A rules-first, targeted LLM workflowenabled near-perfect extraction from finalised structured pulmonary embolism CTreports. Relevance statement A rules-first LLM workflow can automate high-fidelity extraction ofstructured CT findings from finalised radiology reports, enabling scalable,auditable, and more consistent cohort curation for clinical research,registries, and quality improvement. Key Points <UnorderedList Mark="Bullet"> <ItemContent> A hybrid rules-first workflow combining a rule-basedextractor (RBE) with targeted large language model (LLM) salvageachieved the highest overall performance for labelling of pulmonaryembolism CT reports (F1, 0.99; accuracy, 99.8%). </ItemContent> <ItemContent> The top standalone open-weight and proprietary LLMs(Falcon3-10b and GPT-4.1-mini) both exceeded the RBE and, at theupper end of performance, matched a pooled radiologistlabel-transfer baseline. </ItemContent> <ItemContent> The hybrid workflow reduced cohort-curation time from32.2 h for radiologists to 1.0 h while reducing LLM calls by 85.6%,because the LLM was only triggered for rule-failed fields. </ItemContent> </UnorderedList> Graphical Abstract

Who labels best? Radiologists, rules, or large language models for CT reports on pulmonary embolism

期刊论文
出版模式：

开放获取
发表日期：

2026年5月27日

查看全文

Matthias A. Fink,
Arved Bischoff,
Edem Atsiatorme,
Alexander Kremer,
Jonas Kroschke,
Martin Moll,
Patrick Stein,
Veronika Riebl,
Timo Leichenich,
Hans-Ulrich Kauczor,
Kai Schlamp

摘要

Objective

To compare open-weight and proprietary large language models (LLMs),a rule-based extractor (RBE) and radiologists for labelling pulmonary embolismCT reports, and to test whether a hybrid RBE–LLM workflow improveslabelling performance.

Materials and methods

This single-centre retrospective study included structured CTreports from October 2021 to March 2025. Three labelling pipelines wereevaluated: an RBE; a model-agnostic LLM extractor (18 open-weight, four GPT-4variants); and a hybrid pipeline routing only RBE failures to an LLM. Groundtruth was defined at the report-text level by deterministic schema matching forinitially RBE-valid fields and blinded adjudication of RBE-invalid fields by twoattending radiologists. Eight radiologists provided a human baseline. Outcomesincluded F1 scores, accuracy, LLM-based salvage of RBE failures, and labellingtime.

Results

In total, 2,923 reports from 2,923 patients (mean age 66 ± 17 years;1,465 women) were included. Falcon3-10b and GPT-4.1-mini achieved similaritem-level performance (F1 0.98 [95% CI, 0.97–0.98] for both; p = 0.70) and both exceeded the RBE (F1 0.81 [95%CI, 0.80–0.82]; p < 0.001).Salvage of RBE failures was comparable between open-weight and proprietarymodels (88.1% vs 91.9%; p = 0.12). The hybrid RBE–LLM workflowachieved 99.8% accuracy and F1 0.99 (0.98–0.99), exceeding both the RBEand pooled radiologists (F1 0.92 [95% CI, 0.90–0.93]; all p < 0.001).

Conclusion

Schema-constrained open-weight and proprietary LLMs exceededrule-based extraction and, at the upper end of performance, matched a pooledradiologist label-transfer baseline. A rules-first, targeted LLM workflowenabled near-perfect extraction from finalised structured pulmonary embolism CTreports.

Relevance statement

A rules-first LLM workflow can automate high-fidelity extraction ofstructured CT findings from finalised radiology reports, enabling scalable,auditable, and more consistent cohort curation for clinical research,registries, and quality improvement.

Key Points

A hybrid rules-first workflow combining a rule-basedextractor (RBE) with targeted large language model (LLM) salvageachieved the highest overall performance for labelling of pulmonaryembolism CT reports (F1, 0.99; accuracy, 99.8%).

The top standalone open-weight and proprietary LLMs(Falcon3-10b and GPT-4.1-mini) both exceeded the RBE and, at theupper end of performance, matched a pooled radiologistlabel-transfer baseline.

The hybrid workflow reduced cohort-curation time from32.2 h for radiologists to 1.0 h while reducing LLM calls by 85.6%,because the LLM was only triggered for rule-failed fields.

Graphical Abstract