Nearest-Neighbor Language Models (kNN-MT) leverage the contextual representations and next-word predictions of tokens to construct a vector-based database. During the inference stage, this database is utilized to assist the model in predicting the next word, resulting in impressive performance improvements. However, as the volume of data grows, the storage requirements for the vector-based database in kNN-MT continue to increase. Furthermore, the kNN retrieval performed for each predicted token introduces additional latency during the inference stage. To address these limitations, we propose training a lightweight neural network as a substitute for the vector datastore and kNN search. Our approach significantly reduces the storage overhead while maintaining fast inference speed, as demonstrated by our experiments on various translation datasets.