Manual grading of programming assignments is time-consuming and often inconsistent, especially when students submit hundreds of code samples per term. This paper examines the performance of Generative Artificial Intelligence (GenAI) models in automatically categorizing programming errors in student Python code. The study focuses on six common error categories made by novice programmers: syntax, semantic, runtime, typing, control flow, and off-by-one. Seven GenAI models, including GPT-4o, Gemini 2.5, Claude 3.7, and DeepSeekV3, were used to classify these errors in student submissions. The dataset consists of 766 code submissions from Slovak-speaking students, allowing evaluation of model performance in a low-resource language context. Model outputs were compared against teacher annotations using F1 score, precision, recall, and statistical significance tests. Results show that GPT-4o consistently outperforms other models across most error categories, although all models perform poorly in low-frequency categories. These findings support the use of GenAI tools as possible preliminary evaluators in programming education. However, human oversight remains necessary for rare error cases or specific assignments. This paper offers insights into the practical application of GenAI in educational settings, where language and limited resources restrict access to high-end infrastructure.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Evaluating of Generative Artificial Intelligence Models for Automated Categorization of Programming Errors

  • Ľubomír Benko,
  • Janka Pecuchová

摘要

Manual grading of programming assignments is time-consuming and often inconsistent, especially when students submit hundreds of code samples per term. This paper examines the performance of Generative Artificial Intelligence (GenAI) models in automatically categorizing programming errors in student Python code. The study focuses on six common error categories made by novice programmers: syntax, semantic, runtime, typing, control flow, and off-by-one. Seven GenAI models, including GPT-4o, Gemini 2.5, Claude 3.7, and DeepSeekV3, were used to classify these errors in student submissions. The dataset consists of 766 code submissions from Slovak-speaking students, allowing evaluation of model performance in a low-resource language context. Model outputs were compared against teacher annotations using F1 score, precision, recall, and statistical significance tests. Results show that GPT-4o consistently outperforms other models across most error categories, although all models perform poorly in low-frequency categories. These findings support the use of GenAI tools as possible preliminary evaluators in programming education. However, human oversight remains necessary for rare error cases or specific assignments. This paper offers insights into the practical application of GenAI in educational settings, where language and limited resources restrict access to high-end infrastructure.