Combining Dissimilarity Spaces to Improve ANN Search
摘要
In modern data science, high-dimensional datasets with heterogeneous features, outliers and distributed storage are increasingly common. Recent research suggests that alternative distance measures, such as cosine similarity in text mining or embeddings, can outperform traditional metric distances in similarity search tasks. However, most existing exact or approximate methods are designed for metric spaces or centralised environments, limiting their applicability. To bridge this gap, we propose PDASC (Parametrizable Distributed Approximate Similarity Search with Clustering), a distributed indexing algorithm specifically designed for Approximate Nearest Neighbour (ANN) search with arbitrary distance functions. It integrates a lightweight, data-driven pruning strategy based on the empirical cumulative distribution function (ECDF) of distances, combined with local neighbourhood insights from the index structure. This approach enables effective pruning even when global coordination is unfeasible and geometric assumptions do not hold. This project’s broader goal is to explore the impact of using diverse dissimilarity measures in ANN search. As a long-term direction, we aim to investigate the hypothesis that combining multiple dissimilarity representations can yield significantly better performance than relying on a single distance, especially in problems where no clear optimal distance function exists or when each dissimilarity representation emphasises different features of the data.