Scholarlink

Background <p>Systematic literature reviews offer high potential for efficiency gains from artificial intelligence (AI), now integrated into several systematic literature review software platforms. Validation studies show acceptable sensitivity, specificity, and accuracy for AI-assisted systematic literature reviews of clinical trial publications. Unlike trials, economic model publications lack consistency in content, terminology, and structure.</p> Objective <p>We aimed to test the efficiency and accuracy of AI-assisted search, screening, and data extraction when applied to a systematic literature review of economic evaluations.</p> Methods <p>A previously conducted manual systematic literature review of economic evaluations for chronic rhinosinusitis with nasal polyps was replicated using a machine learning-based inclusion prediction model (Robot Screener) and a large language model-based criteria screener (Smart Screener) within Nested Knowledge software, with performance benchmarked against the original human-conducted systematic literature review.</p> Results <p>The AI-generated search retrieved 22/43 (51%) PubMed articles from the original systematic literature review. Accuracy exceeded 95% for title/abstract screening but fell below 80% for full-text screening. Extraction was reliable for high-level model descriptors and general study characteristics, but less so for model structures, health states, outcomes, and distinguishing sensitivity from scenario analyses and complex modeling assumptions for duration of response, discontinuation, surgery, and mortality. Estimated time savings ranged from ~20% (data extraction) to 60% (title/abstract screening and searches), varying by task human validation requirement.</p> Conclusions <p>Artificial intelligence-driven tools performed well for title/abstract screening and general data extraction but were less accurate for full-text screening and interpretation of modeling choices. They can increase systematic literature review efficiency for economic evaluations but fall below the reliability seen for systematic literature reviews of clinical trials.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

A Machine Learning and Large Language Model Tool for Systematic Literature Reviews of Health Economic Evidence: A Validation Study

  • 期刊论文

  • 发表日期：

    2026年7月29日

查看全文
  • Lisa M. Bloudek,
  • Allie B. Cichewicz,
  • Kush Patel,
  • Sean D. Sullivan,
  • Kevin M. Kallmes

摘要

Background

Systematic literature reviews offer high potential for efficiency gains from artificial intelligence (AI), now integrated into several systematic literature review software platforms. Validation studies show acceptable sensitivity, specificity, and accuracy for AI-assisted systematic literature reviews of clinical trial publications. Unlike trials, economic model publications lack consistency in content, terminology, and structure.

Objective

We aimed to test the efficiency and accuracy of AI-assisted search, screening, and data extraction when applied to a systematic literature review of economic evaluations.

Methods

A previously conducted manual systematic literature review of economic evaluations for chronic rhinosinusitis with nasal polyps was replicated using a machine learning-based inclusion prediction model (Robot Screener) and a large language model-based criteria screener (Smart Screener) within Nested Knowledge software, with performance benchmarked against the original human-conducted systematic literature review.

Results

The AI-generated search retrieved 22/43 (51%) PubMed articles from the original systematic literature review. Accuracy exceeded 95% for title/abstract screening but fell below 80% for full-text screening. Extraction was reliable for high-level model descriptors and general study characteristics, but less so for model structures, health states, outcomes, and distinguishing sensitivity from scenario analyses and complex modeling assumptions for duration of response, discontinuation, surgery, and mortality. Estimated time savings ranged from ~20% (data extraction) to 60% (title/abstract screening and searches), varying by task human validation requirement.

Conclusions

Artificial intelligence-driven tools performed well for title/abstract screening and general data extraction but were less accurate for full-text screening and interpretation of modeling choices. They can increase systematic literature review efficiency for economic evaluations but fall below the reliability seen for systematic literature reviews of clinical trials.