Multi-scale Local Region-Based Facial Action Unit Detection with Graph Convolutional Network

Facial action unit (AU) detection is crucial for general facial expression analysis. Different AUs cause facial appearance changes over various regions at different scales, and may interact with each other. However, most existing methods fail to extract the multi-scale feature at local facial region, or consider the AU relationship in the classifiers. In this paper, we propose a novel multi-scale local region-based facial AU detection framework with Graph Convolutional Network (GCN). The proposed framework consists of two parts: multi-scale local region-based (MSLR) feature extraction and AU relationship modeling with GCN. Firstly, to extract the MSLR features, we build the improved AU centers for each AU, and then extract multi-scale feature around the centers with several predefined windows. Secondly, we employ the GCN framework to model the relationship between AUs. Specifically, we build the graph of AUs, then utilize two GCNs to update the MSLR feature and the AU classifiers respectively. Finally, the AU predicted probability is determined by both the multi-scale local feature and the relationship between AUs. Experimental results on two widely used AU detection datasets BP4D and DISFA show that the proposed algorithm outperforms the state-of-the-art methods.