<p>Full-text screening is the major bottleneck of systematic reviews (SRs), particularly in domains such as population health modelling of noncommunicable diseases (NCDs), where decisive eligibility information is scattered across long and heterogeneous full texts. In this methodological study, we introduce a scalable and auditable pipeline that reframes inclusion and exclusion as a fuzzy decision process. We evaluate the approach within the Population Health Modelling Consensus Reporting Network (POPCORN) and benchmark it against statistical and crisp baselines. Articles are parsed into overlapping chunks and embedded with a domain-adapted model; for each criterion (Population, Intervention, Outcome, Study Approach), we compute contrastive similarity (inclusion–exclusion cosine) and a <i>vagueness margin</i>, which a Mamdani fuzzy controller maps into graded inclusion degrees with dynamic thresholds in a multi-label classification setting. A large language model (LLM) judge adjudicates highlighted spans with tertiary labels, confidence scores, and criterion-referenced rationales; when evidence is insufficient, fuzzy membership is attenuated rather than hard-excluded. In a pilot on an all-positive gold set (<InlineEquation ID="IEq1"><EquationSource Format="TEX">\(N{=}16\)</EquationSource></InlineEquation> full texts; <InlineEquation ID="IEq2"><EquationSource Format="TEX">\(M{=}3{,}208\)</EquationSource></InlineEquation> chunks), the fuzzy system achieved document-level recall of 81.25% (Population), 87.50% (Intervention), 87.50% (Outcome), and 75.00% (Study Approach) with 95% Wilson confidence intervals, exceeding statistical baselines (recall 56.25–75.00%) and crisp baselines (recall 43.75–81.25%). Strict “All Criteria” inclusion was reached for 50.00% of articles, compared to 25.00% and 12.50% under the baselines. Cross-model agreement on justifications was 98.27%, and human–machine agreement was 96.07%. A pilot review showed 91% inter-rater agreement (<InlineEquation ID="IEq3"><EquationSource Format="TEX">\(\kappa {=}0.82\)</EquationSource></InlineEquation>) with screening time reduced from <InlineEquation ID="IEq4"><EquationSource Format="TEX">\(\sim\)</EquationSource></InlineEquation>20 minutes to under 1 minute per article at a significantly lower cost. These results suggest that fuzzy logic, combined with contrastive highlighting and explainable LLM adjudication, delivers high recall, stable rationales, and end-to-end traceability. We emphasize that the reported recall figures were computed on an all-positive gold set (no excluded documents), so specificity, ROC/PR curves, and AUC are not defined here; the complete operating-characteristic analysis on a mixed-label corpus will be reported in a forthcoming study.</p>

An auditable pipeline for fuzzy full-text screening in systematic reviews: integrating contrastive semantic highlighting and LLM judgment

期刊论文
出版模式：

开放获取
发表日期：

2026年5月26日

查看全文

Pouria Mortezaagha,
Arya Rahgozar

摘要

Full-text screening is the major bottleneck of systematic reviews (SRs), particularly in domains such as population health modelling of noncommunicable diseases (NCDs), where decisive eligibility information is scattered across long and heterogeneous full texts. In this methodological study, we introduce a scalable and auditable pipeline that reframes inclusion and exclusion as a fuzzy decision process. We evaluate the approach within the Population Health Modelling Consensus Reporting Network (POPCORN) and benchmark it against statistical and crisp baselines. Articles are parsed into overlapping chunks and embedded with a domain-adapted model; for each criterion (Population, Intervention, Outcome, Study Approach), we compute contrastive similarity (inclusion–exclusion cosine) and a vagueness margin, which a Mamdani fuzzy controller maps into graded inclusion degrees with dynamic thresholds in a multi-label classification setting. A large language model (LLM) judge adjudicates highlighted spans with tertiary labels, confidence scores, and criterion-referenced rationales; when evidence is insufficient, fuzzy membership is attenuated rather than hard-excluded. In a pilot on an all-positive gold set (\(N{=}16\) full texts; \(M{=}3{,}208\) chunks), the fuzzy system achieved document-level recall of 81.25% (Population), 87.50% (Intervention), 87.50% (Outcome), and 75.00% (Study Approach) with 95% Wilson confidence intervals, exceeding statistical baselines (recall 56.25–75.00%) and crisp baselines (recall 43.75–81.25%). Strict “All Criteria” inclusion was reached for 50.00% of articles, compared to 25.00% and 12.50% under the baselines. Cross-model agreement on justifications was 98.27%, and human–machine agreement was 96.07%. A pilot review showed 91% inter-rater agreement (\(\kappa {=}0.82\)) with screening time reduced from \(\sim\)20 minutes to under 1 minute per article at a significantly lower cost. These results suggest that fuzzy logic, combined with contrastive highlighting and explainable LLM adjudication, delivers high recall, stable rationales, and end-to-end traceability. We emphasize that the reported recall figures were computed on an all-positive gold set (no excluded documents), so specificity, ROC/PR curves, and AUC are not defined here; the complete operating-characteristic analysis on a mixed-label corpus will be reported in a forthcoming study.