Scholarlink

<p>Prior research has demonstrated the strong reliability and concurrent validity of the comparative judgement (CJ) method for assessing writing quality. However, the factors considered when making these judgements are less investigated, as well as their potential variation across raters with different backgrounds. In Study 1, we asked four participant groups to rate the writing quality of 90 Arabic essays using the CJ method. Regression and correlational analysis showed that expert, crowdsource, and GPT-4o’s CJ ratings exhibited acceptable levels of reliability and concurrent validity, while peers demonstrated only some reliability. In Study 2, we analyzed decision comments left by three groups (experts, crowdsourced workers, peers) after making their judgments. We found evidence for construct validity among experts and crowdsourced workers. These results provide evidence supporting the reliability and validity of the CJ method across three distinct rater groups (experts, crowdsource workers, GPT-4o) and extend its application to a novel assessment context.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Comparative judgment and writing quality: reliability and validity across expert, peer, crowdsourced, and LLM-generated judgments

  • 期刊论文

  • 出版模式：

    开放获取

  • 发表日期：

    2026年5月30日

查看全文
  • Alaa Alzarahni,
  • Hassan Alshumrani,
  • Faten Alarjani,
  • Adel Alfaifi

摘要

Prior research has demonstrated the strong reliability and concurrent validity of the comparative judgement (CJ) method for assessing writing quality. However, the factors considered when making these judgements are less investigated, as well as their potential variation across raters with different backgrounds. In Study 1, we asked four participant groups to rate the writing quality of 90 Arabic essays using the CJ method. Regression and correlational analysis showed that expert, crowdsource, and GPT-4o’s CJ ratings exhibited acceptable levels of reliability and concurrent validity, while peers demonstrated only some reliability. In Study 2, we analyzed decision comments left by three groups (experts, crowdsourced workers, peers) after making their judgments. We found evidence for construct validity among experts and crowdsourced workers. These results provide evidence supporting the reliability and validity of the CJ method across three distinct rater groups (experts, crowdsource workers, GPT-4o) and extend its application to a novel assessment context.