Automatic Short Answer Scoring on an Indian Dataset Using Transformer-Based Language Models

Scoring students’ answers has always been a difficult chore that requires a lot of effort from instructors. The scoring procedure must be quick, and the grades awarded must closely match those assigned by the teacher. The study of ASAS has advanced considerably more since the onset of the epidemic and the shift to distance education. Numerous authors from across the world have offered several solutions to the ASAS problem. This paper aims to show how transformer-based language models can be used for grading short answers on an Indian dataset. In the proposed work, we use transformer-based language models like BERT, GPT-2, XLNet, SBERT, and RoBERTa for the generation of sentence embeddings. These embeddings are generated for the student’s response and the reference response. The resemblance between the student’s response and the reference response is determined using Cosine Similarity. This similarity score is used by the model (Linear Regressor or Ridge Regressor) to generate the predicted score. The baseline dataset, the Mohler dataset, as well as the recently developed Computer Science SAS Dataset in Indian Context (CSDIC), are used to evaluate the model. The Root Mean Square Error and Pearson Correlation are the effectiveness measures (Sanuvala and Fatima in IJRTI 8:1939–1948, 2023) used to assess the model.