Vector similarity search is a crucial component of modern information retrieval systems, in particular using transformer-based embeddings. Nevertheless, scalability and efficiency of these systems is limited owing to the high dimensional latent representations. In this paper, we introduce NashVec, a new game-theoretic approach for learning how to compress the latent space in order to improve the efficiency and semantic relevance of vector search. By casting the compression strategy as a zero-sum game in retrieval accuracy and storage efficiency, we obtain a hidden transformation preserving the semantic similarity but decrease redundancy. We compare NashVec against FAISS, a state-of-the-art vector search library, and show that we achieve much higher average similarity (0.9981 vs. 0.5517) and utility (0.8873 vs 0.5194) at the expense of a slightly slower query time. This trade-off demonstrates the utility of game-theoretic latent compression for high-utility, transformer-based search applications. The approach can be easily integrated with the traditional LLM pipelines while achieving more semantically correct and efficient retrieval. (The code for NashVec can be accessed publicly at: https://github.com/KushagraIsTaken/NashVec ).

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Optimization of Latent-Space Compression Using Game-Theoretic Techniques for Transformer-Based Vector Search

  • Kushagra Agrawal,
  • Nisharg Nargund,
  • Oishani Banerjee

摘要

Vector similarity search is a crucial component of modern information retrieval systems, in particular using transformer-based embeddings. Nevertheless, scalability and efficiency of these systems is limited owing to the high dimensional latent representations. In this paper, we introduce NashVec, a new game-theoretic approach for learning how to compress the latent space in order to improve the efficiency and semantic relevance of vector search. By casting the compression strategy as a zero-sum game in retrieval accuracy and storage efficiency, we obtain a hidden transformation preserving the semantic similarity but decrease redundancy. We compare NashVec against FAISS, a state-of-the-art vector search library, and show that we achieve much higher average similarity (0.9981 vs. 0.5517) and utility (0.8873 vs 0.5194) at the expense of a slightly slower query time. This trade-off demonstrates the utility of game-theoretic latent compression for high-utility, transformer-based search applications. The approach can be easily integrated with the traditional LLM pipelines while achieving more semantically correct and efficient retrieval. (The code for NashVec can be accessed publicly at: https://github.com/KushagraIsTaken/NashVec ).