Facial Expression Recognition in Virtual Reality Simulations

Facial expressions are an important channel for interpersonal communication and comprehension, since people externalize their emotions through a variety of facial expressions. Technology, in particular, deep learning algorithms, can detect and analyze human emotions in real time, which paves the way for advanced user interfaces or adjustable devices and applications. Based on this, the work described in this paper presents a system that identifies three groups of emotions, positive, negative, and neutral, for adjusting user experience in Virtual Reality (VR) games. This, however, introduces an additional challenge in classifying expressions, because of the partial occlusion of the face. To solve this problem, four CNNs were used: VGG-19, ResNet-18, EfficientNet-b1, and Mini-Xception, as well as the ensemble of the three most accurate models (VGG-19, EfficientNet-b1, ResNet-18) via max voting. As expected, this ensemble, which was designated as VERNet, was the most accurate model, with an accuracy of 85.7% without partial occlusion of the face and 82.7% with occlusion. When compared with the accuracy of VGG-19, the results of VERNet only differ by 1% when compared with VGG-19. The minor difference between the results with and without occlusion demonstrated that an outside camera can be a very robust solution for tracking human facial expressions in VR environments.