Distributed Weighted Mahalanobis Distance Spectral Clustering on Spark for Large-Scale Data

Aiming at the problems of large memory occupation and difficulty in adapting to massive scenarios when processing large-scale high-dimensional data with traditional single-machine weighted Mahalanobis distance spectral clustering, this paper proposes a distributed weighted Mahalanobis distance spectral clustering algorithm based on Spark. This algorithm enhances its core performance through three improvements: First, it designs an unsupervised distributed feature weighting mechanism to reduce the interference of redundant and noisy features. Secondly, by combining the local density of the samples and the dynamic scale adjustment of the category size, a multi-scale similarity matrix is constructed to enhance the measurement ability of the weighted Mahalanobis distance for data correlation. Finally, it relies on Spark to achieve data sharding storage, parallel construction of sparse similarity matrices, and distributed feature decomposition, reducing computational complexity and memory overhead, and breaking through the bottleneck of a single machine. Experiments show that the algorithm, while maintaining a clustering accuracy comparable to that of traditional algorithms, improves the efficiency of large-scale data processing and provides an effective solution for clustering large-scale high-dimensional data.