Heterogeneous Face Recognition with Prepended Domain Transformers

Face recognition (FR) has become a very popular method for biometric authentication, thanks to its noncontact nature and high accuracy. State-of-the-art face recognition systems are obtaining human parity even in unconstrained scenarios, thanks to the deep neural network architectures and the large datasets available for training them. However, there are several other types of face imaging modalities such as infrared, thermal, depth, and so on, which can boost the performance of FR systems even further. The main challenge in using these new modalities is the lack of availability of large-scale labeled datasets to train FR models. Heterogeneous face recognition provides a solution to this issue by leveraging the large sets of training data available for visible spectrum data. Heterogeneous face recognition (HFR) involves matching facial images from different domains, such as thermal to visible images (VIS), sketches to visible images, and near-infrared to visible images. This process is especially beneficial for aligning visible spectrum images with those from other modalities. However, HFR poses significant challenges due to the domain gap between the source and target images, compounded by the lack of large-scale, paired heterogeneous face image datasets for training HFR models. In this chapter, we introduce a lightweight and effective method for cross-modality face image matching. The core idea in our approach is to integrate a neural network component, known as the Prepended Domain Transformer (PDT), at the beginning of an existing face recognition (FR) model to bridge the domain gap. By retraining this prepended module with a small number of paired samples in a contrastive learning framework, we achieved state-of-the-art results in various HFR benchmarks. The PDT blocks are versatile and can be retrained for different source-target combinations using our framework. This approach is compatible with any pre-trained FR model due to its architecture-agnostic nature. Additionally, its modular design allows for training with a limited set of paired samples, making it easier for integration and deployment. The source code and protocols for reproducing the results are publicly available.