Cloud computing allows building applications to store and analyze heterogeneous data sources under a pay-as-you-go paradigm. This elasticity leads to significant challenges for Multi-Objective Query Processing (MOQP), requiring Query Execution Plans (QEPs) that balance diverse user priorities, such as execution time, monetary cost, etc. Spark SQL and Spark SQL’s Catalyst Optimizer provide a solution using Cost-based optimization, which can process data from multiple heterogeneous resources. In this solution, they applied the dynamic programming algorithm to pick the best m-way join order. However, when a user would like to expand the cost function to the execution time or monetary cost in a multiple-cloud environment, the optimizer could ignore good plans. To address this problem, we propose a MOQP extension for Spark SQL, leveraging a Non-dominated Sorting Genetic Algorithm based on Grid partitioning (NSGA-G). We validate our proposal with preliminary experimental results.

错误:搜索内容不能为空,请输入英文关键词
错误:关键词超出字数限制,请精简
高级检索

Non-dominated Sorting Genetic Algorithm for Multiple Objectives Query Optimization in Spark SQL

  • Trung-Dung Le

摘要

Cloud computing allows building applications to store and analyze heterogeneous data sources under a pay-as-you-go paradigm. This elasticity leads to significant challenges for Multi-Objective Query Processing (MOQP), requiring Query Execution Plans (QEPs) that balance diverse user priorities, such as execution time, monetary cost, etc. Spark SQL and Spark SQL’s Catalyst Optimizer provide a solution using Cost-based optimization, which can process data from multiple heterogeneous resources. In this solution, they applied the dynamic programming algorithm to pick the best m-way join order. However, when a user would like to expand the cost function to the execution time or monetary cost in a multiple-cloud environment, the optimizer could ignore good plans. To address this problem, we propose a MOQP extension for Spark SQL, leveraging a Non-dominated Sorting Genetic Algorithm based on Grid partitioning (NSGA-G). We validate our proposal with preliminary experimental results.