OPOM: The Ordinal and Parallel Optimization Method of Spark Multi-query Applications
摘要
Optimizing the configuration parameters of Spark applications presents a challenging problem [10, 15] in cloud computing, with the optimization of Spark multi-query applications being the most challenging aspect. Multi-query applications consist of multiple queries, such as TPC-DS [27] and TPC-H [4]. As the amount of data increases, the optimization cost of multi-query applications will increase substantially, making these applications increasingly challenging to optimize. In this study, we propose an Ordinal and Parallel Optimization Method (OPOM). Specifically, we execute queries in parallel and design the Partition Execution Order of queries (ordinal), which can significantly reduce the optimization time cost of multi-query applications. In the benchmarks (TPC-DS and TPC-H) of three data sizes and the Huawei Cloud, OPOM achieved the highest Cost Reduction of optimization time compared to Cherrypick [2], ROBOTune [18], and LOCAT [37], with maximum reductions of 290.84%, 277.26%, and 231.99%, and average reductions of 239.25%, 228.54%, and 194.89%, respectively. The maximum speedups for OPOM, Cherrypick, ROBOTune, and LOCAT are \(12.13 \times \) , \(3.39 \times \) , \( 3.68 \times \) , \(3.73 \times \) , and the corresponding average speedups are \(8.97 \times \) , \(2.89 \times \) , \( 3.00 \times \) , \(3.05 \times \) , respectively.