Scholarlink

<p>Peer review is stressed by rapidly rising submission volumes, leading to deteriorating review quality and increased author dissatisfaction. Here, to address these issues, we developed Review Feedback Agent, a system leveraging multiple large language models to improve review clarity, specificity and actionability by providing automated feedback on vague comments, content misunderstandings and unprofessional remarks to reviewers. We show, through a randomized controlled study at ICLR 2025 with over 20,000 reviews, that 27% of reviewers who received automated feedback updated their reviews, incorporating over 12,000 suggestions. This suggests that many reviewers found the artificial intelligence-generated feedback sufficiently helpful to merit updating their reviews. Blinded evaluation confirmed that revised reviews receiving feedback were more informative. The intervention led to substantially longer reviews (80 additional words among updaters) and increased engagement during rebuttals, with 6% longer author responses and 5.5% longer reviewer replies. This work demonstrates that carefully designed large language model-generated feedback can enhance peer review quality by making reviews more specific and actionable while increasing reviewer–author engagement.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

A large-scale randomized study of large language model feedback in peer review

  • 期刊论文

  • 发表日期：

    2026年2月23日

查看全文
  • Nitya Thakkar,
  • Mert Yuksekgonul,
  • Jake Silberg,
  • Animesh Garg,
  • Nanyun Peng,
  • Fei Sha,
  • Rose Yu,
  • Carl Vondrick,
  • James Zou

摘要

Peer review is stressed by rapidly rising submission volumes, leading to deteriorating review quality and increased author dissatisfaction. Here, to address these issues, we developed Review Feedback Agent, a system leveraging multiple large language models to improve review clarity, specificity and actionability by providing automated feedback on vague comments, content misunderstandings and unprofessional remarks to reviewers. We show, through a randomized controlled study at ICLR 2025 with over 20,000 reviews, that 27% of reviewers who received automated feedback updated their reviews, incorporating over 12,000 suggestions. This suggests that many reviewers found the artificial intelligence-generated feedback sufficiently helpful to merit updating their reviews. Blinded evaluation confirmed that revised reviews receiving feedback were more informative. The intervention led to substantially longer reviews (80 additional words among updaters) and increased engagement during rebuttals, with 6% longer author responses and 5.5% longer reviewer replies. This work demonstrates that carefully designed large language model-generated feedback can enhance peer review quality by making reviews more specific and actionable while increasing reviewer–author engagement.