Spatial Cognition Through Gestural Interfaces: Embodied Play and Learning with Minecraft

Increasingly accessible gesture-based interfaces offer powerful but underexplored potential for facilitating inclusive and engaging learning experiences. In this work, we explore creating and testing a hand and body pose-based gestural control interface for playing Minecraft. This work aims to be a starting point for making Minecraft (and other video games) based learning and play more inclusive and engaging by moving away from just keyboard-mouse controls. We study the viability and user-friendliness of our initial designs through surveys, interviews, and early co-design sessions, shaping gestures that resonate with diverse youth, enriching the learning and assessment of spatial cognition skills among them.