DPSPC: A Density Peak-Based Statistical Parallel Clustering Algorithm for Big Data

Data parallelism is a powerful design paradigm for clustering tasks involving large datasets. However, existing solutions suffer from three problems: (i) using non-identical distribution based partitioning methods may pose the risk of data skew; (ii) frequent communication among the divided data partitions may lead to potential performance degradation; and (iii) unnecessary full computation results in significant computational overhead. In order to address these issues, we propose a density peak-based statistical parallel clustering algorithm for big data (DPSPC). Our sampling-based approach creates equal-sized data blocks with the same statistical measures of clusters, reducing data skew and eliminating inter-block communication. By sampling a subset of blocks for computation, we avoid full computation. The experimental results suggest that the NMI index of the DPSPC algorithm is generally not 10% lower than that of other distributed density peak clustering algorithms, with runtime about one-tenth and the lowest communication volume.