<p>Peer review is the primary mechanism for ensuring the quality of scientific publications. In many conferences, reviewers are required to self-annotate their confidence and provide a recommendation (rating) when evaluating a paper. In cases where the number of submissions is extensive, editors or area chairs rely on these self-rated confidence scores recommendation (rating) to resolve conflicting reviews and borderline cases. However, reviewers may self-assess their confidence in ways that do not align with the content and reasoning presented in their review text, leading to inconsistencies. As a result, these self-assessments may be miscalibrated and fail to accurately reflect the reviewer’s conviction regarding the paper’s merit. Such discrepancies arise when reviewers are uncertain about their evaluations or when their judgments lack sufficient supporting evidence. To address this issue, we introduce <span>ConsistentPeer</span>, a novel knowledge graph-based framework <span>(GraphRAG)</span> designed to systematically assess the consistency between textual review content and assigned scores at multiple granular levels, including word, sentence, and aspect-based analysis. Our approach leverages peer review comments from deep learning and natural language processing conferences to construct an enriched representation of review consistency. This work provides a structured framework to identify and mitigate discrepancies in reviewer self-rated scores, ensuring more justifiable and consistent reviews. By doing so, it enhances the reliability and transparency of the peer review process, particularly in borderline cases.</p>