LLM examiner: automating assessment in informal self-directed e-learning using ChatGPT

Informal e-learning systems often lack structured assessment mechanisms, making it difficult to assess the learning outcomes. This work aims to automate outcome-based assessment through the use of a large language AI model, in particular ChatGPT. Such automation can be of value to educators and developers of educational software, as it tackles the non-trivial task of evaluating educational trajectories from the outcome-based perspective. To achieve this aim, we proposed a system and validated it through a case study and two evaluation stages. In the first stage, we generated 40 assessment questions of various types, 12 of which were approved as high quality. In the second stage, we generated another 45 questions and conducted 5 individual peer evaluation sessions. The most significant automation aspects in guaranteeing the assessment quality were found to be the instructor involvement to monitor the process, the use of a high quality custom knowledge base, and formulation of the correct prompt instructions on the basis of the learning outcome statements.