Automatic Generation of Multiple-Choice Questions for CS0 and CS1 Curricula Using Large Language Models

In the context of increasing attention to formative assessment in universities, Multiple Choice Question (MCQ) has become a vital assessment form for CS0 and CS1 courses due to its advantages of rapid assessment, which has brought about a significant demand for MCQ exercises. However, creating many MCQs takes time and effort for teachers. A practical method is to use large language models (LLMs) to generate MCQs automatically, but when dealing with specific domain problems, the model results may need to be more reliable. This article designs a set of prompt chains to improve the performance of LLM in education. Based on this design, we developed EduCS, which is based on GPT-3.5 and can automatically generate complete MCQs according to the CS0/CS1 course outline. To evaluate the quality of MCQs generated by EduCS, we established a set of evaluation metrics from four aspects about the three components of MCQ and the complete MCQ, and based on this, we utilized expert scoring. The experimental results indicate that while the generated questions require teacher verification before being delivered to students, they show great potential in terms of quality. The EduCS system demonstrates the ability to generate complete MCQs that can complement formative and summative assessments for students at different levels. The EduCS has great promise value in the formative assessment of CS education.