Deep Learning Based Audio-Visual Emotion Recognition in a Smart Learning Environment

The main goal of this work is to develop a method to monitor the emotional state of students and teachers during the study process based on voice and facial expressions, that can be applied in a smart learning environment. In this paper we create a multimodal emotion detection model based on voice and facial expression features using convolutional neural network (CNN) models. We describe the implementation of the created emotion detection model into the learning process as a web application to monitor the emotional state of students and teachers in a smart learning environment. In this work we compare three types of emotion detection models: models based on audio and facial features separately and for both features taken together and test the performance of these models in the simulation of study process. To evaluate and analyze the models’ performances k-fold cross-validation is applied and classification accuracy, weighted F1 score, and confusion matrix are computed. The application developed in this study allows to identify the overall emotional background of the learning environment, determine the emotional state of students and academic staff during the learning process in near real-time.