This study investigates the grading tendencies and effectiveness of large language models (LLMs) compared to human evaluators in assessing special-purpose texts, specifically first-year Computer Science research proposals. The dataset comprised 81 proposals authored by 243 students and graded by six instructors using a standardized rubric. Grades assigned by two LLMs, ChatGPT 3.5 and Gemini, were compared to human grades using regression analysis. The findings reveal that ChatGPT 3.5 aligned more closely with human graders, achieving a 53.09% match, while Gemini demonstrated weaker alignment at 27.16%. ChatGPT 3.5 displayed a statistically significant predictive relationship with human grades, supported by a coefficient of 0.741 and a p-value of 0.039. Conversely, Gemini’s coefficient of 0.038 and p-value of 0.825 indicated limited predictive power. Human graders assigned the highest proportion of A grades, with no failing grades, while both LLMs were stricter, particularly Gemini, which awarded no A grades and assigned failing marks to three proposals. Regression analyses confirmed that ChatGPT 3.5 explained 92.3% of the variance in human grading with an adjusted R-squared value of 0.846, indicating strong alignment. In contrast, Gemini exhibited significant divergence, failing to replicate human grading behavior effectively. These results highlight the potential of ChatGPT 3.5 for automated grading of special-purpose texts, while emphasizing the limitations of Gemini. This research underscores the importance of prompt design and alignment when deploying LLMs for grading tasks and suggests further exploration to enhance their consistency and reliability.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Grading the Future: A Comparative Analysis of Large Language Models and Human Instructors in Evaluating First-Year Computer Science Research Proposals

  • Georgy Gelvanovsky

摘要

This study investigates the grading tendencies and effectiveness of large language models (LLMs) compared to human evaluators in assessing special-purpose texts, specifically first-year Computer Science research proposals. The dataset comprised 81 proposals authored by 243 students and graded by six instructors using a standardized rubric. Grades assigned by two LLMs, ChatGPT 3.5 and Gemini, were compared to human grades using regression analysis. The findings reveal that ChatGPT 3.5 aligned more closely with human graders, achieving a 53.09% match, while Gemini demonstrated weaker alignment at 27.16%. ChatGPT 3.5 displayed a statistically significant predictive relationship with human grades, supported by a coefficient of 0.741 and a p-value of 0.039. Conversely, Gemini’s coefficient of 0.038 and p-value of 0.825 indicated limited predictive power. Human graders assigned the highest proportion of A grades, with no failing grades, while both LLMs were stricter, particularly Gemini, which awarded no A grades and assigned failing marks to three proposals. Regression analyses confirmed that ChatGPT 3.5 explained 92.3% of the variance in human grading with an adjusted R-squared value of 0.846, indicating strong alignment. In contrast, Gemini exhibited significant divergence, failing to replicate human grading behavior effectively. These results highlight the potential of ChatGPT 3.5 for automated grading of special-purpose texts, while emphasizing the limitations of Gemini. This research underscores the importance of prompt design and alignment when deploying LLMs for grading tasks and suggests further exploration to enhance their consistency and reliability.