Towards generalizing facial action unit recognition for real-world applications

Facial action unit (AU) recognition models are used in various applications, as they provide a defined and explainable understanding of facial expressions. Many current AU recognition models are limited in application performance due to a lack of generalization across identity, lighting, head pose, and background. Those existing models, despite having high reported metrics, have limited usefulness in real-world applications. One cause of the lack of generalization is the limited variation, as the most popular and standard datasets, BP4D and DISFA, contain only 41 or 27 individuals, respectively. To address this issue, we use a face reenactment method to increase the training set size by over 125 times the original number of identities. We also use a cross-corpus loss to increase the number of predicted AUs from 12 to 15, resulting in denser predictions and a more accurate AU recognition model. Our model achieves the highest average performance on the combined BP4D and DISFA datasets. To further emphasize the usefulness of our model and show its potential for real-world applications, we compare various open-source AU recognition models on three applications: emotion recognition, drowsiness detection, and engagement classification. These applications were selected to demonstrate a range of potential applications that focus exclusively on facial behavior, each of which the AU recognition model has never seen before. Our model outperforms all other open-source AU recognition models by up to 6.3 percentage points on the three applications.