Scholarlink

<p>This explanatory sequential mixed-methods research examined Generative Artificial Intelligence (GenAI)- and student-authored cause and effect essays to reveal reliability, consistency, rater agreement between different scoring methods and to document decision-making strategies for identifying authorship. The participants (<i>n</i> = 107) in a teacher education program scored cause and effect essays through AI-generated rubrics, a standardized IELTS rubric, and a holistic general impression, and identified authorship of AI-generated and student-written essays. Data were analyzed through logistic regression, Generalizability Theory, and thematic analysis to examine detection patterns and pedagogical decision-making and reasoning processes. The results showed variation of scoring across rubric types which also interacted with rater type, rater expertise, and detection of authorship in complex ways. The instructor-pre-service teacher agreement was highest when using holistic scoring; AI-pre-service teacher agreement was highest for the IELTS rubric, and AI-instructor agreement was relatively high for the AI-generated rubric. The results showed that rubric type, rater expertise, and detection of authorship interact in complex ways. We revealed misdetections and a critical quality attribution paradox where sophisticated student writing was associated with AI, while polished AI-generated essay was identified as student-written. We documented multi-dimensional pedagogical reasoning strategies that moved from initial intuitive impressions to textual analysis that included indicators of originality and authorial stance. The results indicated that reliability of using GenAI for AES and teacher scoring may be at risk when authorship cannot be correctly identified. This study contributed to arguments about challenges to consistency and the need for control mechanisms and specialized training.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

GenAI for automated essay scoring: A turing test of rubrics, rater agreement, and authorship in L2 writing assessment

  • 期刊论文

  • 发表日期：

    2026年7月1日

查看全文
  • Sibel Söğüt,
  • Serap Büyükkıdık

摘要

This explanatory sequential mixed-methods research examined Generative Artificial Intelligence (GenAI)- and student-authored cause and effect essays to reveal reliability, consistency, rater agreement between different scoring methods and to document decision-making strategies for identifying authorship. The participants (n = 107) in a teacher education program scored cause and effect essays through AI-generated rubrics, a standardized IELTS rubric, and a holistic general impression, and identified authorship of AI-generated and student-written essays. Data were analyzed through logistic regression, Generalizability Theory, and thematic analysis to examine detection patterns and pedagogical decision-making and reasoning processes. The results showed variation of scoring across rubric types which also interacted with rater type, rater expertise, and detection of authorship in complex ways. The instructor-pre-service teacher agreement was highest when using holistic scoring; AI-pre-service teacher agreement was highest for the IELTS rubric, and AI-instructor agreement was relatively high for the AI-generated rubric. The results showed that rubric type, rater expertise, and detection of authorship interact in complex ways. We revealed misdetections and a critical quality attribution paradox where sophisticated student writing was associated with AI, while polished AI-generated essay was identified as student-written. We documented multi-dimensional pedagogical reasoning strategies that moved from initial intuitive impressions to textual analysis that included indicators of originality and authorial stance. The results indicated that reliability of using GenAI for AES and teacher scoring may be at risk when authorship cannot be correctly identified. This study contributed to arguments about challenges to consistency and the need for control mechanisms and specialized training.