Landmark-Assisted Facial Action Unit Detection with Optimal Attention and Contrastive Learning

In this paper, we propose a weakly-supervised algorithm for facial action unit (AU) detection in the wild, which combines basic facial features and attention-based landmark features as well as contrastive learning to improve the performance of AU detection. Firstly, the backbone is a weakly-supervised algorithm since AU datasets in the wild are scarce and the utilization of other public datasets can capture robust basic facial features and landmark features. Secondly, we explore and select the optimal attention-based landmark encoder to capture facial landmark features that have been shown highly related to AUs. Then, we combine basic facial features and attention-based landmark features for AU detection. Finally, we propose a weighted multi-label contrastive loss function for the further improvement of AU detection. Extensive experiments on RFAU and BP4D demonstrate that our method outperforms or is comparable with state-of-the-art weakly-supervised and supervised AU detection methods.