RAISE: Reasoning Agent for Interactive SQL Exploration

2025-06-02

AI Summary

Existing natural language-to-SQL systems suffer from low accuracy due to inadequate database schema understanding and ambiguous or incomplete user queries. Method: This paper proposes the first end-to-end, unified interactive reasoning framework—replacing traditional multi-stage pipelines with a large language model–driven autonomous agent architecture. It integrates dynamic schema awareness, execution-feedback-driven reflection, multi-turn candidate generation, and diversity-enhancement strategies to emulate the human “hypothesize–verify–reflect” database exploration process. Crucially, it introduces a deep, scalable test-time computational resource allocation mechanism to handle under-specified queries. Contribution/Results: On the BIRD benchmark, our approach achieves an 11.7-percentage-point improvement in execution accuracy (44.8% → 56.5%), with Best-of-8 accuracy reaching 81.8%—on par with state-of-the-art methods—while substantially reducing system engineering complexity.

Technology Category

• Search and Optimization: Learning to Search
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Recent advances in large language models (LLMs) have propelled research in natural language interfaces to databases. However, most state-of-the-art text-to- SQL systems still depend on complex, multi-stage pipelines. This work proposes a novel agentic framework that unifies schema linking, query generation, and itera- tive refinement within a single, end-to-end component. By leveraging the intrinsic reasoning abilities of LLMs, our method emulates how humans answer questions when working with unfamiliar databases: understanding the data by formulating hypotheses, running dynamic queries to validate them, reasoning over the results, and revising outputs based on observed results. Crucially, our approach intro- duces a new strategy for scaling test-time computation in text-to-SQL: we scale the depth of interactive database exploration and reflection. This shift enables the model to allocate computation dynamically to better understand the data, especially useful in ambiguous and underspecified scenarios. Our experiments show that it improved the Execution Accuracy (EX) from 44.8% to 56.5% on the challenging BIRD dataset using DeepSeek-R1-Distill-Llama-70B. Fur- thermore, when equipped with steps to add more diversity to the answers, our agent achieves a Best-of-N accuracy of 81.8% with 8 rounds of candidate gener- ation, rivaling the 82.79% achieved by the top-ranked published solution, while reducing engineering complexity. These findings position our unified framework as a promising alternative for building natural language interfaces to databases.

Problem

• Unifying schema linking, query generation, and refinement in text-to-SQL
• Scaling test-time computation via interactive database exploration
• Improving execution accuracy in ambiguous database query scenarios

Innovation

• Unified agentic framework for text-to-SQL tasks
• Dynamic query validation and iterative refinement
• Scaling test-time computation via interactive exploration