This study investigates using a chatbot powered by a pre-trained neural network language model to automate the grading of short-answer questions in higher education. Implementing digital learning technologies, such as chatbots, offers innovative solutions to enhance the assessment process by reducing the manual grading burden on academic staff, particularly in large classes. In this research, a chatbot was employed to grade three variations of short-answer questions—description, comparison, and listing—in an introduction to networking course within the computer science curriculum. A sample of 228 near-graduation information science students from a research-intensive university participated, with short answers manually scored by the course instructor using a rubric that provided scores ranging from 0 to 5. A subset of these graded answers was used to fine-tune the chatbot’s language model, creating a training set. The chatbot’s grading performance was evaluated by comparing its ratings to those of human graders using quadratic-weighted kappa (QWKappa) to determine agreement levels. Results demonstrated that the chatbot achieved good inter-rater agreement across all question types, particularly excelling in grading description and comparison questions. This study highlights the potential of chatbots as a practical tool for automated grading, improving efficiency and accuracy in digital assessments.