Universal Facial Landmark Detection by Landmark-Clustering Relation-Reasoning Transformer

Current facial landmark detection methods typically train separate models for individual datasets, limiting their ability to fully leverage labeled information across datasets and reducing the landmark detection accuracy. Additionally, these models continue to face challenges with large poses and partial occlusions, as they struggle to accurately infer facial structures under these conditions. To tackle these challenges, this paper proposes a universal facial landmark detection framework called Landmark-Clustering Relation-Reasoning Transformer (LCRRT), which enables an all-in-one model that allows testing across different datasets without changing any parameters. LCRRT explores the dependencies between image tokens by a Transformer architecture, and models facial landmark detection as a progressively optimized landmark coordinate prediction process. To achieve facial landmark detection across datasets, a novel Hierarchical Facial Region Clustering (HFRC) model is designed to yield region tokens through a parameterized clustering method for learning a universal facial structure. Additionally, a Landmark Relation Reasoning (LRR) loss is introduced to guide the clustering process to further improve the universal feature learning. This is the first study to explore an all-in-one facial landmark detection model by modeling a universal facial structure. Extensive experimental results on popular benchmark datasets demonstrate that our LCRRT achieves comparable performance with the state-of-the-art methods. The code is available at https://github.com/bleucon/LCRRT.