Resolving Memory Challenges in Cluster Computing Systems Via Stratified Asymptotic Sampling for Big Data Classification
摘要
In cluster computing systems, memory limitations have become a critical bottleneck for big data classification. We propose a Stratified Asymptotic Sampling (SAS) method based on Hadoop Distributed File System (HDFS) storage mechanism to address this issue. SAS treats HDFS data blocks as sampling strata, implementing intra-block random sampling and block-level progressive learning strategy. A voxel-based d-dimensional Kolmogorov-Smirnov (vdKS) test serves as a stopping criterion to ensure sampling consistency. The method reduces memory demands by progressive processing, enabling fast sampling. The designed secondary indexing system stores only block numbers and line numbers of the sampled data, creating an efficient data retrieval mechanism. Experiments demonstrate that SAS significantly improves sampling efficiency when data size far exceeds cluster memory capacity, showing excellent performance by using a progressive enhancement classification framework with a Memory-Adaptive Progressive Ensemble (MAPE) strategy.