Scalable Approximate Nearest Neighbor Search with PCA-Augmented HNSW in Vector Databases

The rapid growth of data-driven applications, such as LLMs and recommendation systems, has intensified the need for efficient retrieval over high-dimensional unstructured data. Traditional databases struggle with large-scale indexing and querying, motivating the rise of vector databases as key infrastructure for similarity search in AI workloads. Leveraging approximate nearest neighbor (ANN) algorithms, these systems balance retrieval accuracy with computational and memory constraints. However, state-of-the-art ANN methods, such as HNSW, still face scalability challenges due to high computational costs and memory overhead. This paper presents an extension of the HNSW algorithm that integrates Principal Component Analysis (PCA) for dimensionality reduction. Unlike prior approaches, the method preserves the original data distribution using a post-retrieval re-ranking of results from the reduced-dimensional subspace. Experiments on standard benchmarks demonstrate significant reductions in memory usage and index construction time, while maintaining high recall and query responsiveness. Performance analysis shows up to 81% improvement in indexing efficiency and up to 170% gain in query throughput on average for high-dimensional datasets.