Vector Quantization Improvement Algorithm for Controlling Average Distortion in the Context of Big Data

Big data has penetrated into every corner of life, in which vector quantization algorithms are widely used. However, due to the huge amount of data and continuous update in the context of big data, the classical vector quantization algorithm needs to re-operate after the data is updated and the codebook cannot guarantee the uniqueness. This results in a huge amount of computation and affects the training of other subsequent models. In this paper, we propose to control the uniformity of the codebook with the average distortion degree, and on this basis, we keep the original codebook and filter the data to train the new version. The experimental results prove that the method can be used in all dimensions of sample data, which greatly reduces the computational effort and repetitive operations.