Incorporating Timing Data into PISA Population Modeling
摘要
The 2015 cycle of PISA was among the first attempts to collect process data, including response time (RT), in an international large-scale assessment. In particular, RT data have the potential to be incorporated, along with many other non-cognitive variables, at the population modeling stage, contributing to the improved accuracy of plausible values and less biased secondary analyses statistics. In this chapter, we provide an overview of the RT variables provided to the public in PISA, and we present the pilot study addressing two research questions: (1) how to handle the item-level RT data to incorporate them appropriately into the population modeling, and (2) whether or not the inclusion RT information contributes to improving the reporting of proficiency scores. For the pilot study, we describe the procedures and present the results using the data collected in selected countries during the PISA 2015 cycle, which eventually guided the operational decisions for the PISA 2018 population modeling. For the first research question, we present one practical and straightforward method (among many different approaches) to derive person-level characteristics out of the item-by-person RT variables recorded in the log files. For the second research question, we compare the empirical analyses results, depending on the inclusion or exclusion of derived RT variables in the generation of plausible values. We conclude the chapter with discussion about limitations, implications, and suggestions for future studies.