DataGenX: Generating Synthetic Relational Data from Annotated SQL Schemas
摘要
Synthetic data generation is essential for database development, allowing for the simulation, benchmarking and testing of the customer workload in scenarios where real data is unavailable due to privacy, scale or accessibility constraints. In practice, developers often have only the database schema and a handful of statistics, such as table cardinalities, value distributions, number of distinct values, TOP-N values, and range-based histograms. Despite this common setting, existing tools fall short: benchmark-specific generators (e.g., TPC-H, TPC-C) are rigid and tied to fixed schemas; UI-based tools like Mockaroo ignore relational semantics; and model-based approaches such as SDV require access to real data to learn patterns. We present DataGenX (part of the TiDB toolchain), a lightweight and flexible synthetic data generator designed for this under-served but common scenario. DataGenX operates directly on annotated SQL DDL, where comments are used to specify column-level statistics and inter-table relationships (e.g., foreign key join cardinalities). Then it generates synthetic data that conforms to these annotations, producing realistic and consistent datasets without requiring access to real data. DataGenX also supports multi-threaded row generation, where each thread produces a disjoint set of rows and writes to a separate file, enabling scalable and efficient data generation on multi-core systems. To validate the fidelity of DataGenX, we reimplemented the TPC-C and TPC-H schemas using annotated DDL and generated data using DataGenX. We compared the resulting datasets to those produced by the official TPC data generators and DataGenX that achieved fidelity 99%, accurately reproducing distributions, key relationships and table sizes. This demonstrates DataGenX’s capability to simulate realistic workloads purely from schema-level specifications and a small set of statistical annotations.