<p>Peer review is the primary mechanism for ensuring the quality and integrity of scholarly publications. Reviewers are often asked to explicitly report their level of confidence when evaluating a manuscript, particularly in conference reviewing. In many cases, especially for lengthy or complex submissions, editors and area chairs rely on these self-rated confidence scores to resolve conflicting reviews and make decisions in borderline cases. However, reviewers’ self-annotations of confidence do not always align with the impressions conveyed by the review text itself, leading to potential inconsistencies. As a result, such self-reported confidence is frequently miscalibrated and may fail to accurately reflect a reviewer’s true conviction regarding the merits of a paper. Review texts often contain <i>uncertainty-related linguistic cues</i>, such as hedging expressions (e.g., <i>maybe, seems, might</i>), which can indicate ambiguity, limited supporting evidence, or cautious reasoning. Editors and area chairs must implicitly account for these cues when interpreting reviews and making final decisions. In this work, we propose a generic attention-based framework to computationally estimate reviewer confidence directly from review text by modeling uncertainty-related linguistic cues including hedging expressions and expressions of conviction within a unified attention-based architecture. We further evaluate strong baseline approaches based on Bidirectional Encoder Representations from Transformers (BERT) and domain-specific SciBERT representations. Our experimental results demonstrate that the proposed attention-based model, grounded in SciBERT representations, achieves encouraging performance compared to strong baselines across multiple evaluation metrics. Importantly, we do not treat hedging as a direct proxy for epistemic uncertainty; instead, it is modeled as a probabilistic linguistic cue whose contribution to confidence estimation is learned contextually through supervised training.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

An uncertainty and conviction-aware attention model for automatically estimating reviewer confidence from peer review texts

  • Prabhat Kumar Bharti

摘要

Peer review is the primary mechanism for ensuring the quality and integrity of scholarly publications. Reviewers are often asked to explicitly report their level of confidence when evaluating a manuscript, particularly in conference reviewing. In many cases, especially for lengthy or complex submissions, editors and area chairs rely on these self-rated confidence scores to resolve conflicting reviews and make decisions in borderline cases. However, reviewers’ self-annotations of confidence do not always align with the impressions conveyed by the review text itself, leading to potential inconsistencies. As a result, such self-reported confidence is frequently miscalibrated and may fail to accurately reflect a reviewer’s true conviction regarding the merits of a paper. Review texts often contain uncertainty-related linguistic cues, such as hedging expressions (e.g., maybe, seems, might), which can indicate ambiguity, limited supporting evidence, or cautious reasoning. Editors and area chairs must implicitly account for these cues when interpreting reviews and making final decisions. In this work, we propose a generic attention-based framework to computationally estimate reviewer confidence directly from review text by modeling uncertainty-related linguistic cues including hedging expressions and expressions of conviction within a unified attention-based architecture. We further evaluate strong baseline approaches based on Bidirectional Encoder Representations from Transformers (BERT) and domain-specific SciBERT representations. Our experimental results demonstrate that the proposed attention-based model, grounded in SciBERT representations, achieves encouraging performance compared to strong baselines across multiple evaluation metrics. Importantly, we do not treat hedging as a direct proxy for epistemic uncertainty; instead, it is modeled as a probabilistic linguistic cue whose contribution to confidence estimation is learned contextually through supervised training.