Scholarlink

<p>Systematic reviews are essential but labor-intensive. We evaluated LLM-assisted literature screening and drafting in two hepatology topics: carvedilol in compensated cirrhosis and anticoagulation in portal vein thrombosis. For each topic, we searched PubMed, Cochrane, and EMBASE. A few-shot prompt with explicit inclusion/exclusion criteria was used to screen titles and abstracts, with results compared to manual review. Included studies were then processed using a retrieval-augmented LLM to generate ten automated systematic review and meta-analysis drafts per topic, which were evaluated by a separate judge LLM for PRISMA 2020 compliance against human reviews. Screening performance: After deduplication (703 and 370 records), LLM-assisted screening showed high agreement with manual review (sensitivity 86–93%, specificity 96–99%) while reducing screening time to 3 and 2 h versus 62 and 30 h manually. Drafting performance: RAG-enabled LLMs generated structured manuscripts with variable PRISMA 2020 compliance: 100% for titles, 91% for introductions, 75–80% for methods, and 68–75% for results, with downstream weaknesses in abstracts and discussions (&lt;65%). LLM-based PRISMA scoring closely matched human review (ICC ≈ 0.90). LLM-assisted screening was highly accurate, reducing workload by &gt;90%, but automated drafting was reliable mainly for titles and introductions, requiring human oversight to prevent errors and hallucinations.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Closing the screening gap but not the writing gap: a two-topic evaluation of LLMs for systematic reviews and meta-analyses in hepatology

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年6月22日

查看全文
  • Yuntao Zou,
  • Iris Kim,
  • Nan Gao,
  • Michelle Li,
  • Mi-Ok Kim,
  • Jin Ge

摘要

Systematic reviews are essential but labor-intensive. We evaluated LLM-assisted literature screening and drafting in two hepatology topics: carvedilol in compensated cirrhosis and anticoagulation in portal vein thrombosis. For each topic, we searched PubMed, Cochrane, and EMBASE. A few-shot prompt with explicit inclusion/exclusion criteria was used to screen titles and abstracts, with results compared to manual review. Included studies were then processed using a retrieval-augmented LLM to generate ten automated systematic review and meta-analysis drafts per topic, which were evaluated by a separate judge LLM for PRISMA 2020 compliance against human reviews. Screening performance: After deduplication (703 and 370 records), LLM-assisted screening showed high agreement with manual review (sensitivity 86–93%, specificity 96–99%) while reducing screening time to 3 and 2 h versus 62 and 30 h manually. Drafting performance: RAG-enabled LLMs generated structured manuscripts with variable PRISMA 2020 compliance: 100% for titles, 91% for introductions, 75–80% for methods, and 68–75% for results, with downstream weaknesses in abstracts and discussions (<65%). LLM-based PRISMA scoring closely matched human review (ICC ≈ 0.90). LLM-assisted screening was highly accurate, reducing workload by >90%, but automated drafting was reliable mainly for titles and introductions, requiring human oversight to prevent errors and hallucinations.