<p>Data stream clustering (DSC) is one of the most significant and widely studied research directions in the field of data mining. However, for the processing of large-scale data streams, most existing methods suffer from slow speed, insufficient memory, and lack of detection and response mechanisms for concept drift. In this paper, a novel Large-Scale Stream K-measn method (LS<sup>2</sup>K-means) is proposed. By first introducing Product-Quantized Codes into the framework of stream clustering methods, memory space consumption is reduced through the dimensionality reduction of data. Additionally, a new similarity measurement method is introduced, greatly improving the efficiency of distance calculation. A concept drift detection and response mechanism is constructed. By comparing the consistency of clustering results, concept drift can be quickly detected, and a backtracking mechanism is utilized to respond to concept drift promptly, effectively improving the algorithm’s performance. The experimental results on six real datasets show that the proposed method can effectively deal with concept drift in data streams. Especially on large-scale datasets, the execution time of the proposed method exceeds that of the comparison methods.</p>

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Large-Scale Stream k-means based on Product-Quantized codes

  • Yuqing Hang,
  • Hongwei Yin,
  • Wenjun Hu,
  • Longfei Zhong,
  • Yuzhou Ni

摘要

Data stream clustering (DSC) is one of the most significant and widely studied research directions in the field of data mining. However, for the processing of large-scale data streams, most existing methods suffer from slow speed, insufficient memory, and lack of detection and response mechanisms for concept drift. In this paper, a novel Large-Scale Stream K-measn method (LS2K-means) is proposed. By first introducing Product-Quantized Codes into the framework of stream clustering methods, memory space consumption is reduced through the dimensionality reduction of data. Additionally, a new similarity measurement method is introduced, greatly improving the efficiency of distance calculation. A concept drift detection and response mechanism is constructed. By comparing the consistency of clustering results, concept drift can be quickly detected, and a backtracking mechanism is utilized to respond to concept drift promptly, effectively improving the algorithm’s performance. The experimental results on six real datasets show that the proposed method can effectively deal with concept drift in data streams. Especially on large-scale datasets, the execution time of the proposed method exceeds that of the comparison methods.