A Multimodal Machine Learning Approach for Attentiveness Detection via Gaze and Emotion Analysis
摘要
The research investigates how to effectively measure student engagement during in-person or online learning where observation is often unreliable and challenging. This study proposes a novel approach that utilizes deep learning and computer vision to study student attention: Gaze Net for gaze estimation, emotion recognition models for facial expression classification, and YOLOv8 for fast detection of faces. By developing a non-invasive, real-time approach to measure attention, this study has shown that the method supports measurement of attention based on gaze and emotional patterns, and clearly outperforms traditional approaches, by providing an automated, continuous measure. The process of utilizing methods for assessing emotional indicators and visually-focused indicators involves face imaging, data preparation, and subsequent use of trained deep learning models. The methodology developed provides better performance during comparison of conditions in real-time environments demonstrating greater scalability and robustness and useful for both online and offline learning environments. Additionally, one use of this methodology is to provide real-time feedback to support intervention and effectiveness of treatment knowledge.