Reviewer 2 Must Be Stopped: Transformer-Based Approaches for Predicting Paper Acceptance

Peer review is a fundamental process in assessing the quality of papers for publication in academic settings. However, peer review is reported to be often highly subjective leading to undesirable decisions being made. Studies demonstrated that two independent review boards would accept and reject the same paper 50% of the time. In an attempt to support paper authors in assessing their chances of getting their paper accepted, previous work proposed traditional machine learning-based approaches (e.g., decision trees with handcrafted features) to predict whether a paper will be accepted or rejected. To the best of our knowledge, state-of-the-art pre-trained language models, i.e., transformers, have not been explored for this problem as of yet. To address this gap, we developed transformer models for the above-mentioned classification problem, and compared their performance with support vector machine (SVM) and multilayer perceptron (MLP) models. Specifically, we fine-tuned BERT, RoBERTa and Longformer transformer models on the ASAP-Review dataset, and compared their performance with traditional machine learning-based models trained on handcrafted features. Among the transformer models, Longformer obtained the highest accuracy of 76.92% on the test set. Overall, MLP obtained the best accuracy of 78.37%. We believe that our proposed classification system can support conference participants, helping them in assessing their papers before submission.