SQLBarber: A System Leveraging Large Language Models to Generate Customized and Realistic SQL Workloads

2025-07-08

AI Summary

Acquiring realistic SQL queries is challenging, and existing generation methods lack customization and fail to satisfy practical constraints. Method: This paper proposes the first fully automated SQL workload generation framework, integrating a declarative natural language interface, an LLM-driven self-correcting pipeline, and a Bayesian optimizer. Users specify constraints via plain text, enabling automatic synthesis of high-fidelity, large-scale SQL queries that precisely match target cost distributions—including cardinalities and execution plan costs. Contribution/Results: Compared to conventional approaches, our framework accelerates generation by 1–3 orders of magnitude. We construct and open-source ten benchmarks grounded in real-world statistics from Snowflake and Amazon Redshift. Experimental results demonstrate significantly improved alignment with query feature distributions. To our knowledge, this is the first end-to-end, high-accuracy, and customizable SQL workload generation solution.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Search and Optimization: Sampling/Simulation-based Search
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Web data generation and simulation

Abstract

Database research and development often require a large number of SQL queries for benchmarking purposes. However, acquiring real-world SQL queries is challenging due to privacy concerns, and existing SQL generation methods are limited in customization and in satisfying realistic constraints. To address this issue, we present SQLBarber, a system based on Large Language Models (LLMs) to generate customized and realistic SQL workloads. SQLBarber (i) eliminates the need for users to manually craft SQL templates in advance, while providing the flexibility to accept natural language specifications to constrain SQL templates, (ii) scales efficiently to generate large volumes of queries matching any user-defined cost distribution (e.g., cardinality and execution plan cost), and (iii) uses execution statistics from Amazon Redshift and Snowflake to derive SQL template specifications and query cost distributions that reflect real-world query characteristics. SQLBarber introduces (i) a declarative interface for users to effortlessly generate customized SQL templates, (ii) an LLM-powered pipeline augmented with a self-correction module that profiles, refines, and prunes SQL templates based on query costs, and (iii) a Bayesian Optimizer to efficiently explore different predicate values and identify a set of queries that satisfy the target cost distribution. We construct and open-source ten benchmarks of varying difficulty levels and target query cost distributions based on real-world statistics from Snowflake and Amazon Redshift. Extensive experiments on these benchmarks show that SQLBarber is the only system that can generate customized SQL templates. It reduces query generation time by one to three orders of magnitude, and significantly improves alignment with the target cost distribution, compared with existing methods.

Problem

• Generates customized SQL workloads using LLMs for benchmarking
• Overcomes privacy and realism limitations in SQL query acquisition
• Optimizes query cost distribution alignment with real-world characteristics

Innovation

• LLM-based SQL workload generation without manual templates
• Natural language specifications for SQL constraints
• Bayesian Optimizer for cost-aligned query generation