WorthyPar: A Workload-Aware Data Hybrid Partitioning Advisor with Deep Reinforcement Learning

Data partitioning physically divides tables or databases to minimize I/O and maximize query processing performance. Designing subtle partitioning strategies for OLAP workloads is an important and challenging task. However, existing workload-aware partitioning strategies lack flexibility and fine-grained partitioning strategies lack adaptability. To address these limitations, we propose WorthyPar. To our knowledge, this is the first attempt to achieve self-driving hybrid partitioning relying on DRL. Specifically, we first demonstrate that the hybrid partitioning problem is NP-hard and formulate it as a Markov Decision Process (MDP) to train an automatic partitioning advisor. Subsequently, we propose a workload prediction model to forecast future workloads, and a performance analysis model to assess the performance of hybrid partitioning strategies without actual partitioning in the database. Extensive experiments demonstrate that WorthyPar can achieve up to 75% reduction in time.