A Data Skew Greedy Optimization Strategy in Spark Heterogeneous Clusters
摘要
In distributed computing frameworks, data skew is one of the main factors affecting performance improvement. The essence of this problem is that there are differences in the computing load of partition tasks, resulting in node load imbalance, especially in heterogeneous scenarios. This paper proposes a data skew greedy optimization strategy (HSDSGO) based on heterogeneous Spark to solve the data skew problem, which comprehensively considers differences in node performance and data locality. Firstly, we use an improved reservoir sampling algorithm to sample the input data in the Map phase, obtaining the key distribution in the entire dataset. Secondly, the capacity size of mapping partition tasks is determined based on the evaluation of node computational capabilities. Finally, the data is partitioned using a best-fit decreasing algorithm. Experimental results demonstrate that the HSDSGO algorithm not only reduces task execution time but also maximizes the utilization of cluster resources.