Scholarlink

Purpose <p>This study evaluates Large Language Models as automated analyzers for detecting students’ erroneous mental models in programming education, addressing the scalability barrier of labor-intensive manual analysis. Building upon preliminary GPT-4 findings, this investigation extends to GPT-5, assessing generational improvements in reliability, error patterns, and computational feasibility.</p> Methods <p>GPT-4 and GPT-5 analyzed 1,500 student responses to 24 C++ programming questions, yielding 5,300 binary judgments for 27 hypothesized mental models compared against expert faculty evaluations. Inter-rater agreement was quantified through percentage agreement and Cohen’s Kappa, with confusion matrix analysis revealing systematic error patterns.</p> Results <p>GPT-5 achieved 82.7% raw agreement with faculty (vs. GPT-4’s 66.9%) and Cohen’s Kappa of <InlineEquation ID="IEq1"> <EquationSource Format="TEX">\(\kappa \)</EquationSource> </InlineEquation> = 0.38 (vs. 0.18), though below the 0.60-−0.80 inter-human threshold. Both models exhibited false positive bias, with GPT-5’s reduced rate (11.6% vs. 23.1%) driving improvement. GPT-5 demonstrated 7.6-fold speed improvement and 34% cost reduction ($12.51 vs. $19.06).</p> Conclusion <p>Current LLMs achieve production-ready status within hybrid human-AI frameworks. GPT-5’s fair-to-moderate reliability combined with cost-performance improvements ($50 per 100-student cohort) establishes feasibility at educational scales. However, reliability insufficient for autonomous deployment supports hybrid configurations wherein LLMs serve as first-pass analyzers subject to systematic faculty review.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Evaluating GPT as automated analyzer for detecting students’ erroneous mental models in programming education

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年3月10日

查看全文
  • Francisco J. Gallego-Durán,
  • Patricia Compañ-Rosique,
  • Carlos J. Villagrá-Arnedo

摘要

Purpose

This study evaluates Large Language Models as automated analyzers for detecting students’ erroneous mental models in programming education, addressing the scalability barrier of labor-intensive manual analysis. Building upon preliminary GPT-4 findings, this investigation extends to GPT-5, assessing generational improvements in reliability, error patterns, and computational feasibility.

Methods

GPT-4 and GPT-5 analyzed 1,500 student responses to 24 C++ programming questions, yielding 5,300 binary judgments for 27 hypothesized mental models compared against expert faculty evaluations. Inter-rater agreement was quantified through percentage agreement and Cohen’s Kappa, with confusion matrix analysis revealing systematic error patterns.

Results

GPT-5 achieved 82.7% raw agreement with faculty (vs. GPT-4’s 66.9%) and Cohen’s Kappa of \(\kappa \) = 0.38 (vs. 0.18), though below the 0.60-−0.80 inter-human threshold. Both models exhibited false positive bias, with GPT-5’s reduced rate (11.6% vs. 23.1%) driving improvement. GPT-5 demonstrated 7.6-fold speed improvement and 34% cost reduction ($12.51 vs. $19.06).

Conclusion

Current LLMs achieve production-ready status within hybrid human-AI frameworks. GPT-5’s fair-to-moderate reliability combined with cost-performance improvements ($50 per 100-student cohort) establishes feasibility at educational scales. However, reliability insufficient for autonomous deployment supports hybrid configurations wherein LLMs serve as first-pass analyzers subject to systematic faculty review.