Scholarlink

<p>Selecting an appropriate Diagnostic Classification Model (DCM) is critical for obtaining accurate feedback on student skills. For Bayesian DCMs, the standard Deviance Information Criterion (DIC) is limited as it summarizes the posterior distribution with a single point estimate, and it favors complex models. This study evaluates two alternatives that utilize the full posterior distribution: the Widely Applicable Information Criterion (WAIC) and Pareto-Smoothed Importance Sampling Leave-One-Out Cross-Validation (PSIS-LOO). A simulation study varied generating models, item quality, prior information level, sample size, and estimation models with Q-matrix specification to compare performance based on true model recovery and the classification accuracy of selected models. Results showed WAIC and PSIS-LOO were more often superior to DIC at identifying the correct model. Critically, models selected by WAIC and PSIS-LOO yielded more accurate student attribute and profile classifications, particularly with small samples and uninformative prior. This study provides strong evidence that WAIC and PSIS-LOO are preferable to DIC, leading to more reliable and valid diagnostic information.</p>

错误：搜索内容不能为空，请输入英文关键词
错误：关键词超出字数限制，请精简
高级检索

Evaluating WAIC and PSIS-LOO for bayesian diagnostic classification model selection

  • 期刊论文

  • 发表日期：

    2026年3月9日

查看全文
  • Ae Kyong Jung,
  • Jonathan Templin

摘要

Selecting an appropriate Diagnostic Classification Model (DCM) is critical for obtaining accurate feedback on student skills. For Bayesian DCMs, the standard Deviance Information Criterion (DIC) is limited as it summarizes the posterior distribution with a single point estimate, and it favors complex models. This study evaluates two alternatives that utilize the full posterior distribution: the Widely Applicable Information Criterion (WAIC) and Pareto-Smoothed Importance Sampling Leave-One-Out Cross-Validation (PSIS-LOO). A simulation study varied generating models, item quality, prior information level, sample size, and estimation models with Q-matrix specification to compare performance based on true model recovery and the classification accuracy of selected models. Results showed WAIC and PSIS-LOO were more often superior to DIC at identifying the correct model. Critically, models selected by WAIC and PSIS-LOO yielded more accurate student attribute and profile classifications, particularly with small samples and uninformative prior. This study provides strong evidence that WAIC and PSIS-LOO are preferable to DIC, leading to more reliable and valid diagnostic information.