Rationalization Models for Text-to-SQL

2025-02-10

AI Summary

To address the low execution accuracy and poor interpretability of medium-to-high-complexity queries in text-to-SQL tasks, this paper proposes a Chain-of-Thought (CoT) rationalization framework tailored for Text-to-SQL. Methodologically, it introduces a novel paradigm combining human-annotated guidance for large language model prompt generation with iterative dynamic few-shot knowledge distillation, enabling efficient construction of high-quality, scalable CoT data annotated with intermediate SQL queries and natural-language explanations. A lightweight rationalization model is then supervisedly trained on this data. Evaluated on the BIRD benchmark, the approach significantly improves execution accuracy—especially for medium-to-high-complexity queries—while generating clear, faithful, step-by-step reasoning traces. Thus, it achieves a favorable trade-off between performance and interpretability, advancing both practical utility and model transparency in Text-to-SQL systems.

Technology Category

• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Data Mining & Knowledge Management: Intelligent Query Processing

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

We introduce a framework for generating Chain-of-Thought (CoT) rationales to enhance text-to-SQL model fine-tuning. These rationales consist of intermediate SQL statements and explanations, serving as incremental steps toward constructing the final SQL query. The process begins with manually annotating a small set of examples, which are then used to prompt a large language model in an iterative, dynamic few-shot knowledge distillation procedure from a teacher model. A rationalization model is subsequently trained on the validated decomposed queries, enabling extensive synthetic CoT annotations for text-to-SQL datasets. To evaluate the approach, we fine-tune small language models with and without these rationales on the BIRD dataset. Results indicate that step-by-step query generation improves execution accuracy, especially for moderately and highly complex queries, while also enhancing explainability.

Problem

• Enhance text-to-SQL model fine-tuning
• Generate Chain-of-Thought rationales
• Improve SQL query execution accuracy

Innovation

• Chain-of-Thought rationales generation
• Iterative few-shot knowledge distillation
• Enhanced text-to-SQL execution accuracy