Scholarlink

<p>Systematic reviews are essential for evidence-based research, yet the traditional screening process is time-consuming and difficult to scale. Human-only screening can introduce inconsistency, while fully automated approaches employing Large Language Models often lack the contextual judgement required for complex decisions. To address this, we introduce a crowd-based screening methodology that integrates human expertise with adaptive machine learning. The methods have been applied in the context of a large EU project where experts from 27 collaborating partners jointly screened 5842 papers across eleven disease topics related to patient-generated health data in a span of 2 days. Post-processing played a central role in ensuring data quality, including topic reallocation, targeted full-text verification, and noisy-label filtering. This Screenathon resulted in 487 records being labeled as relevant and 6,463 records as irrelevant. The number of records screened per participant ranged from 3 to 2496, with a mean of 216.4 records per screener (<i>SE</i> = 95.19). Exploratory analyses using survey results indicated increased trust in AI-assisted systematic reviewing after the event, along with generally positive evaluations of usability. The current Screenathon demonstrates that crowdsourced human–AI collaboration requires thoughtful training and calibration, together with strong post-processing safeguards.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Screenathon 2.0: human–AI collaborative screening applied to patient-generated health data

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年3月22日

查看全文
  • Jonas Bergmann,
  • Tiago Azzi,
  • Rutger Neeleman,
  • Kianush Monschau,
  • Berke Yazan,
  • Elena Jalsovec,
  • Emily Westerbeek,
  • Felix Weijdema,
  • Jonathan de Bruin,
  • Qixiang Fang,
  • Rens van de Schoot

摘要

Systematic reviews are essential for evidence-based research, yet the traditional screening process is time-consuming and difficult to scale. Human-only screening can introduce inconsistency, while fully automated approaches employing Large Language Models often lack the contextual judgement required for complex decisions. To address this, we introduce a crowd-based screening methodology that integrates human expertise with adaptive machine learning. The methods have been applied in the context of a large EU project where experts from 27 collaborating partners jointly screened 5842 papers across eleven disease topics related to patient-generated health data in a span of 2 days. Post-processing played a central role in ensuring data quality, including topic reallocation, targeted full-text verification, and noisy-label filtering. This Screenathon resulted in 487 records being labeled as relevant and 6,463 records as irrelevant. The number of records screened per participant ranged from 3 to 2496, with a mean of 216.4 records per screener (SE = 95.19). Exploratory analyses using survey results indicated increased trust in AI-assisted systematic reviewing after the event, along with generally positive evaluations of usability. The current Screenathon demonstrates that crowdsourced human–AI collaboration requires thoughtful training and calibration, together with strong post-processing safeguards.