Generating adversarial SQL queries for evaluating cardinality estimators
摘要
Query cardinality estimators are typically assessed by relying on pre-defined queries, generated from random or expert-derived templates, that can suffer from low query coverage and hence lead to poorly representative results. As an alternative, we propose a framework called Advice for generating adversarial queries that works by ascending local P-error gradients, where P-error quantifies the plan error between the golden plan generated under true cardinalities and the query plan generated under a cardinality estimator. However, P-error is expensive to calculate because it requires enumerating the true cardinality of every subquery. To address this challenge, we propose a GNN-based technique to estimate the direction of the gradient in order to speed up exploration. To keep up with the high exploration rate during candidate verification, we design a scheduling technique that dynamically prioritizes queries which are more likely to be truly adversarial. But as with any gradient-based technique, encountering local maxima requires costly restarts. To reduce the risk of restarts and lessen their impact, we propose a template and bucketing technique for efficiently finding high-quality initial starting queries, along with a predictive measure which we call “future benefit” for avoiding local maxima in the first place. Experiments on three datasets show that Advice outperforms the baseline approaches in both query generation quality and generation efficiency, and that the generated queries provide a practical stress-testing workload that facilitates analysis of cardinality estimators.