Large-Scale Stream k-means based on Product-Quantized codes
摘要
Data stream clustering (DSC) is one of the most significant and widely studied research directions in the field of data mining. However, for the processing of large-scale data streams, most existing methods suffer from slow speed, insufficient memory, and lack of detection and response mechanisms for concept drift. In this paper, a novel Large-Scale Stream K-measn method (LS2K-means) is proposed. By first introducing Product-Quantized Codes into the framework of stream clustering methods, memory space consumption is reduced through the dimensionality reduction of data. Additionally, a new similarity measurement method is introduced, greatly improving the efficiency of distance calculation. A concept drift detection and response mechanism is constructed. By comparing the consistency of clustering results, concept drift can be quickly detected, and a backtracking mechanism is utilized to respond to concept drift promptly, effectively improving the algorithm’s performance. The experimental results on six real datasets show that the proposed method can effectively deal with concept drift in data streams. Especially on large-scale datasets, the execution time of the proposed method exceeds that of the comparison methods.