Optimization of Latent-Space Compression Using Game-Theoretic Techniques for Transformer-Based Vector Search
摘要
Vector similarity search is a crucial component of modern information retrieval systems, in particular using transformer-based embeddings. Nevertheless, scalability and efficiency of these systems is limited owing to the high dimensional latent representations. In this paper, we introduce NashVec, a new game-theoretic approach for learning how to compress the latent space in order to improve the efficiency and semantic relevance of vector search. By casting the compression strategy as a zero-sum game in retrieval accuracy and storage efficiency, we obtain a hidden transformation preserving the semantic similarity but decrease redundancy. We compare NashVec against FAISS, a state-of-the-art vector search library, and show that we achieve much higher average similarity (0.9981 vs. 0.5517) and utility (0.8873 vs 0.5194) at the expense of a slightly slower query time. This trade-off demonstrates the utility of game-theoretic latent compression for high-utility, transformer-based search applications. The approach can be easily integrated with the traditional LLM pipelines while achieving more semantically correct and efficient retrieval. (The code for NashVec can be accessed publicly at: https://github.com/KushagraIsTaken/NashVec ).