Analyzing Test-Taking Behaviors Through Process Data Using the IRT Explanatory Model for Guessing
摘要
Guessing behavior is a known problem in international large-scale assessments (ILSAs), and it can impact the assessment’s reliability and validity by introducing random error and bias. Testing and accounting for such biases is essential in ensuring the accuracy and comparability of test scores and secondary analysis results. This chapter proposes a flexible item response theory (IRT) modeling approach for guessing behavior: the IRT Explanatory Model for Guessing (IRT-EMG). The IRT-EMG combines mixture models with response time and other process data in-formation (number of actions) from computer-based ILSAs. It presents an extension of the grade of membership response time model for detecting guessing behaviors proposed by Pokropek (Journal of Educational and Behavioral Statistics 41: 300–325, 2016) and is based on the grade of membership (GoM) model (Erosheva, 2002 Grade of membership and latent structure models with application to disability survey data (Doctoral dissertation). Carnegie Mellon University)) and the HYBRID model (Yamamoto, HYBRID model of IRT and latent class models (ETS Research Report RR-89-41). Educational Testing Service, 1989; Yamamoto & Everson, Modeling the effects of test length and test time on parameter estimation using the HYBRID model. In J. Rost (Ed.), Applications of latent trait and latent class models in the social sciences (pp. 89–98), Waxmann, 1997). Modeling the effects of test length and test time on parameter estimation using the HYBRID model. We show that IRT-EMG is a generalization of the classical 3PL-IRT model and that the 3PL-IRT model could be presented as a restricted case of the IRT-EMG. This chapter describes the features and advantages of the IRT-EMG model and illustrates its application on data from the Programme for the International Assessment of Adult Competencies (PIAAC). Results show how information about different characteristics of items, respondents, and process data (response time, number of actions) might help detect different types of guessing behaviors and identify items that are particularly prone to guessing.