This chapter illustrates how a machine supported coding system (MSCS) was developed and implemented for automatic scoring of constructed responses in the Programme for International Student Assessment (PISA) and presents its performance. As an international large-scale survey, PISA 2018 collected data from students across more than 80 countries and in about 50 languages in the computer-based mode. Conceptually, the MSCS can be viewed as a supervised learning of natural language data that is generalizable across multiple languages to be used for PISA data. Without extracting language-dependent features, the MSCS utilizes the exact-matching method and automatically applies a verified label to a subset of incoming uncoded responses, which guarantees the highest accuracy in the prediction. In the PISA 2018 assessment cycle, the MSCS has shown to alleviate scoring burden as much as 25% on average across all participating countries and across domains. This chapter starts with the motivation for developing the MSCS to benefit a large-scale setting with multiple languages of assessment. We also present the findings from an analysis of PISA 2018 data and explore possible avenues for continuing to improve.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Automatic Scoring of Constructed Responses in PISA Using the Machine-Supported Coding System

  • Hyo Jeong Shin,
  • Kentaro Yamamoto,
  • Qiwei He,
  • Matthias von Davier

摘要

This chapter illustrates how a machine supported coding system (MSCS) was developed and implemented for automatic scoring of constructed responses in the Programme for International Student Assessment (PISA) and presents its performance. As an international large-scale survey, PISA 2018 collected data from students across more than 80 countries and in about 50 languages in the computer-based mode. Conceptually, the MSCS can be viewed as a supervised learning of natural language data that is generalizable across multiple languages to be used for PISA data. Without extracting language-dependent features, the MSCS utilizes the exact-matching method and automatically applies a verified label to a subset of incoming uncoded responses, which guarantees the highest accuracy in the prediction. In the PISA 2018 assessment cycle, the MSCS has shown to alleviate scoring burden as much as 25% on average across all participating countries and across domains. This chapter starts with the motivation for developing the MSCS to benefit a large-scale setting with multiple languages of assessment. We also present the findings from an analysis of PISA 2018 data and explore possible avenues for continuing to improve.