SPRAG: building and benchmarking a Short Programming-Related Answer Grading dataset

Automated Short Answer Grading (ASAG) is a widely explored application of NLP in the domain of education. While much research focuses on natural language responses, this work introduces the creation and evaluation of the Short Programming-Related Answer Grading dataset (SPRAG). The corpus comprises questions and answers extracted from programming subjects, involving symbols, keywords, and no explicit grammar. Our key contributions include the curation of this manually annotated code-mixed short answer dataset, along with guidelines for corpus annotation, ensuring substantial Inter-Annotator Agreement. This work also explores the dataset and provides initial analysis of the dataset. In the context of auto-grading, we evaluate a range of pre-trained sentence-transformer models by fine-tuning them with the SPRAG corpus for binary and multi-class classification tasks. Specifically, the binary classification task aims to discern between two classes (0 and 1), while the multi-class classification task involves assigning grades on a scale of 0–5 to each answer. Our best fine-tuned model achieves an accuracy of 86.16% for binary classification and 56.11% for multi-class classification. In conclusion, our research contributes a valuable resource in the form of the SPRAG dataset, catering to the ASAG and NLP communities alike. We are committed to promoting further research by making the dataset and accompanying code freely available to the public.