Automated essay grading is a complex task that combines educational assessment, language understanding, and predictive modeling. It requires systems to interpret linguistic nuance, evaluate discourse-level structure, and provide trait-specific feedback. Despite decades of research, existing approaches often fall short in consistency and fairness, particularly when applied to essays written by English Language Learners (ELLs). In this study, we benchmark a set of pretrained transformer models on a standardized essay grading dataset, evaluating their performance across six analytic writing traits using the Mean Columnwise Root Mean Squared Error (MCRMSE) metric. Beyond accuracy comparisons, this work introduces a comparative perspective that explores whether performance on widely used NLP benchmarks such as SQuAD and MNLI aligns with effectiveness in essay scoring. Our findings suggest that high benchmark performance does not necessarily predict success in downstream educational tasks. This opens new directions for evaluating model suitability in under-benchmarked domains through cross-task alignment.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Investigating the Relevance of NLP Benchmarks for Automated Essay Grading: A Multi-trait Evaluation of Pre-trained Transformers

  • Abdelaziz Qassi,
  • Ibrahim Riza Hallac,
  • Hasan Ogul

摘要

Automated essay grading is a complex task that combines educational assessment, language understanding, and predictive modeling. It requires systems to interpret linguistic nuance, evaluate discourse-level structure, and provide trait-specific feedback. Despite decades of research, existing approaches often fall short in consistency and fairness, particularly when applied to essays written by English Language Learners (ELLs). In this study, we benchmark a set of pretrained transformer models on a standardized essay grading dataset, evaluating their performance across six analytic writing traits using the Mean Columnwise Root Mean Squared Error (MCRMSE) metric. Beyond accuracy comparisons, this work introduces a comparative perspective that explores whether performance on widely used NLP benchmarks such as SQuAD and MNLI aligns with effectiveness in essay scoring. Our findings suggest that high benchmark performance does not necessarily predict success in downstream educational tasks. This opens new directions for evaluating model suitability in under-benchmarked domains through cross-task alignment.