Similarity-based face image retrieval using sparsely embedded deep features and binary code learning

Human face retrieval has long been established as one of the most interesting research topics in computer vision. With the recent development of deep learning, many researchers have addressed this problem by building deep hashing models to learn binary code from face images, while performing face retrieval as a classification task. Nevertheless, the performance is still unsatisfactory since these models are incapable of handling inter-class variation between multiple persons, as we need to make a class label for each identity. In this backdrop, we propose in this paper an effective deep learning-based framework for face image retrieval. The key to our framework is mainly based on the matching of face pairs, where a two-stream network, named \(\chi Net+\chi Match\) $\chi Net+\chi Match$ , is designed to learn similarities in terms of person identity. Such similarities are investigated by embedding both deep local representation via face components, and deep global face representation via the whole face image. Since the similarities captured over face components are supposed to diversify due to variation in pose, expression and occlusion, we also introduce a Sparse Score Fusion layer that learns automatically the weight of each component according to its contribution to face matching. To allow fast retrieval, we farther propose a method that generates binary codes corresponding to the groups of similar faces through the hierarchical k-means, where the path down binary tree is exploited as a binary code for indexing. The final retrieval is then conducted within a privileged subset of images in the database. Our experiments on different challenging datasets show that our approach obtains outstanding results while outperforming most existing methods.