LLMSQL: Upgrading WikiSQL for the LLM Era of Text-to-SQL

2025-09-27

AI Summary

WikiSQL suffers from pervasive annotation flaws—including case inconsistency, type mismatches, syntactic errors, and unanswerable questions—rendering it inadequate for modern Text-to-SQL research with large language models (LLMs). Method: We introduce LLMSQL, the first LLM-optimized re-engineered benchmark derived from WikiSQL. It discards the original pointer-network dependency, provides standardized natural-language questions paired with complete, executable SQL queries, and employs automated error classification, cleaning, and re-annotation to enhance data consistency. Contribution/Results: We conduct systematic evaluation across state-of-the-art open-weight LLMs—including Gemma 3, LLaMA 3.2, Mistral 7B, Qwen 2.5, Phi-3.5 Mini, and DeepSeek R1. Experiments demonstrate that LLMSQL significantly improves generation reliability and evaluation robustness. It establishes a high-quality, scalable, and LLM-friendly benchmark for Text-to-SQL, enabling more rigorous and reproducible research in the LLM era.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Question Answering
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Converting natural language questions into SQL queries (Text-to-SQL) enables non-expert users to interact with relational databases and has long been a central task for natural language interfaces to data. While the WikiSQL dataset played a key role in early NL2SQL research, its usage has declined due to structural and annotation issues, including case sensitivity inconsistencies, data type mismatches, syntax errors, and unanswered questions. We present LLMSQL, a systematic revision and transformation of WikiSQL designed for the LLM era. We classify these errors and implement automated methods for cleaning and re-annotation. To assess the impact of these improvements, we evaluated multiple large language models (LLMs), including Gemma 3, LLaMA 3.2, Mistral 7B, gpt-oss 20B, Phi-3.5 Mini, Qwen 2.5, OpenAI o4-mini, DeepSeek R1 and others. Rather than serving as an update, LLMSQL is introduced as an LLM-ready benchmark: unlike the original WikiSQL, tailored for pointer-network models selecting tokens from input, LLMSQL provides clean natural language questions and full SQL queries as plain text, enabling straightforward generation and evaluation for modern natural language-to-SQL models.

Problem

• Addressing structural and annotation issues in WikiSQL dataset
• Converting natural language questions into accurate SQL queries
• Creating an LLM-ready benchmark for modern Text-to-SQL models

Innovation

• Automated cleaning and re-annotation methods
• Transformed dataset for LLM text generation
• Provides clean natural language and SQL pairs