Integrating Data from Multiple Sources in Evaluation Studies of Educational Games: An Application of Cross-Classified Item Response Theory Modeling
摘要
It is crucial to evaluate the purported instructional benefits of educational games while developing tools to integrate insights from the rich interactions they facilitate into reliable systems of measurement of learning. Existing game-based evaluation studies use statistical tools that rarely integrate information from different sources. These sources include traditional test items assessing skills or knowledge targeted by the game, fine-grained gameplay process data (moment-to-moment records of learners’ game-based interactions), background surveys measuring attributes of the individuals or their contexts (e.g. schools), and information on game or game level design features. We present a new application of a type of psychometric model (cross-classified item response theory modeling) to analyze gameplay and assessment data collected from a large-scale game-based randomized controlled trial. This application (a) jointly models data collected from multiple sources, allowing for a more holistic evaluation of the game’s instructional effect; (b) quantifies changes in individuals’ educational outcomes; (c) relates changes to gameplay behaviors and patterns. The application can be extended to include individuals’ background and game level design information (explanatory predictors) in model equations to answer substantive questions. We demonstrate the advantages of our application over three other approaches to analyzing gameplay and assessment data. We also note the implications for using game-based analytic results to inform learning and instruction.