SEED: Enhancing Text-to-SQL Performance and Practical Usability Through Automatic Evidence Generation

2025-06-09

AI Summary

Current Text-to-SQL research relies heavily on human-annotated evidence—e.g., in the BIRD benchmark—contradicting the original goal of “non-expert friendliness” and suffering from incomplete or erroneous manual evidence. This work identifies a critical gap: significant performance degradation and practical limitations in evidence-free settings. We propose SEED, the first end-to-end automatic evidence generation framework that requires no human annotation. SEED jointly leverages database schema parsing, metadata semantic understanding, and value distribution analysis, employing lightweight prompt engineering to synthesize highly relevant domain knowledge. Evaluated on BIRD and Spider, SEED substantially improves SQL accuracy under evidence-free conditions—outperforming human-evidence baselines in several configurations—while enhancing model robustness and cross-domain generalization. By bridging the gap between benchmark evaluation and real-world deployment, SEED advances practical, scalable Text-to-SQL systems.

Technology Category

• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Data Mining & Knowledge Management: Intelligent Query Processing
• Knowledge Representation and Reasoning: Automated Reasoning and Theorem Proving

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Text-to-SQL enables non-experts to retrieve data from databases by converting natural language queries into SQL. However, state-of-the-art text-to-SQL studies rely on the BIRD dataset, which assumes that evidence is provided along with questions. Although BIRD facilitates research advancements, it assumes that users have expertise and domain knowledge, contradicting the fundamental goal of text-to-SQL. In addition, human-generated evidence in BIRD contains defects, including missing or erroneous evidence, which affects model performance. To address this issue, we propose SEED (System for Evidence Extraction and Domain knowledge generation), an approach that automatically generates evidence to improve performance and practical usability in real-world scenarios. SEED systematically analyzes database schema, description files, and values to extract relevant information. We evaluated SEED on BIRD and Spider, demonstrating that it significantly improves SQL generation accuracy in the no-evidence scenario, and in some cases, even outperforms the setting where BIRD evidence is provided. Our results highlight that SEED-generated evidence not only bridges the gap between research and real-world deployment but also improves the adaptability and robustness of text-to-SQL models. Our code is available at https://github.com/felix01189/SEED

Problem

• Automatically generates evidence for text-to-SQL queries
• Improves SQL accuracy without relying on human-provided evidence
• Enhances real-world usability and robustness of text-to-SQL models

Innovation

• Automatically generates evidence for text-to-SQL
• Analyzes schema, description files, and values
• Improves SQL accuracy without human evidence