Learning to Find Good Hash Functions for Embeddings
摘要
Our context of interest is locality sensitive hashing over embeddings generated by neural networks, in particular binary locality sensitive functions. These functions map objects in the embedding space to a binary value. To be locality sensitive, similar objects must have a higher probability of generating a hash collision than randomly selected objects. In this paper, we investigate the use of locality sensitive hash functions to address the ANN problem. We demonstrate that whilst uniform spaces exhibit good locality sensitivity properties, spaces derived from the output of neural networks do not. We describe a method to dynamically select instances of good locality sensitive hash functions on a per-query basis, based on a sample of the dataset. Using this analysis, we demonstrate a mechanism to efficiently search embedding spaces with linear preprocessing cost, allowing fast build times for large datasets.