Rater severity drift (RSD) is a critical challenge in performance assessments, especially when only a single rater is available. Many-facet Rasch approaches typically require multiple raters, limiting their applicability in classroom contexts where one rater scores repeated assessments over time. This study evaluates three hierarchical Bayesian models—modified formulations of Uto (2023), Usami (2010), and Yokouchi et al. (2024)—for estimating RSD from sparse longitudinal data. Using oral-summary scores from 64 Japanese university students across four time points, we assessed convergence and compared models with the widely applicable information criterion (WAIC) and leave-one-out cross-validation (LOO), posterior predictive p-values (PPP), and root-mean-square error (RMSE). All models converged satisfactorily. The Uto variant achieved the most favorable information-criterion values and the lowest RMSE in this dataset, whereas the Usami variant produced the smoothest and most interpretable severity trajectories—useful for monitoring and communication. The Yokouchi variant prioritizes simplicity, with some loss of precision. These findings show that hierarchical Bayesian frameworks can quantify temporal shifts in rater behavior under single-rater, low-information conditions, supporting quality assurance and rater development in resource-constrained educational settings.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Modeling Rater Severity Drift in Single-Rater Performance Assessments Using Bayesian Hierarchical Methods

  • Yuichiro Yokouchi,
  • Kuangzhe Xu,
  • Shuichi Takaki,
  • Haruhiko Mitsunaga

摘要

Rater severity drift (RSD) is a critical challenge in performance assessments, especially when only a single rater is available. Many-facet Rasch approaches typically require multiple raters, limiting their applicability in classroom contexts where one rater scores repeated assessments over time. This study evaluates three hierarchical Bayesian models—modified formulations of Uto (2023), Usami (2010), and Yokouchi et al. (2024)—for estimating RSD from sparse longitudinal data. Using oral-summary scores from 64 Japanese university students across four time points, we assessed convergence and compared models with the widely applicable information criterion (WAIC) and leave-one-out cross-validation (LOO), posterior predictive p-values (PPP), and root-mean-square error (RMSE). All models converged satisfactorily. The Uto variant achieved the most favorable information-criterion values and the lowest RMSE in this dataset, whereas the Usami variant produced the smoothest and most interpretable severity trajectories—useful for monitoring and communication. The Yokouchi variant prioritizes simplicity, with some loss of precision. These findings show that hierarchical Bayesian frameworks can quantify temporal shifts in rater behavior under single-rater, low-information conditions, supporting quality assurance and rater development in resource-constrained educational settings.