ABISS: Evaluating Text-to-SQL Systems Through Agent Interaction

2026-07-25

AI Summary

Current Text-to-SQL systems struggle with ambiguous or unanswerable user questions and lack evaluation frameworks grounded in realistic interactive scenarios. This work proposes the ABIS framework, which establishes a dynamic multi-turn dialogue simulation environment. By integrating a unified question taxonomy with a two-stage, multi-agent collaborative generation pipeline, ABIS constructs a high-quality dataset encompassing eight categories of ambiguous and unanswerable questions. The framework further incorporates category consistency validation and a style-aware user simulator. Experiments on ABIS-BIRD and ABIS-Spider reveal significant bottlenecks in subcategory recognition and post-clarification SQL generation. While providing ground-truth category labels substantially improves execution accuracy and feedback quality, overall performance on ambiguous questions remains limited and necessitates further improvement.

Abstract

Large Language Models (LLMs) demonstrate high performance on curated Text-to-SQL benchmarks; nevertheless, real-world users frequently pose ambiguous or unanswerable questions that current systems handle poorly. Three interconnected gaps hinder progress: incomplete taxonomies, realistic benchmark generation for real-world settings, and static user interaction. We address all of the above issues through three contributions: (1) a unified taxonomy of 8 categories covering ambiguous and unanswerable questions; (2) a multi-agent generation pipeline with a two-stage process (NLQ generation followed by SQL grounding) and an explicit Category Conformance validation stage, producing questions from arbitrary databases validated by a council of local open-source models; and (3) ABISS (Ambiguity Benchmark using Interaction-Simulated Sessions), a dynamic simulation environment where Text-to-SQL agents interact with style-aware simulated users across multi-turn dialogues. Experiments with eight open-source models on ABISS-BIRD and ABISS-Spider reveal two fundamental bottlenecks. The first is subcategory classification: models detect that a question is problematic yet struggle to pinpoint the specific subcategory. The second is clarification-conditioned SQL generation: even after receiving useful user information, models often still fail in the final resolution step. Providing the ground truth category yields large gains in both execution and feedback across both datasets, yet ambiguous-question execution remains low even under oracle category labels. We release our code for data generation and benchmark on GitHub (https://github.com/giosullutrone/ABISS-Evaluating-Text-to-SQL-Systems-Through-Agent-Interaction).

Problem

• Text-to-SQL
• ambiguous questions
• unanswerable questions
• benchmarking
• user interaction

Innovation

• Text-to-SQL
• Ambiguity Handling
• Multi-agent Simulation
• Interactive Evaluation
• Benchmark Generation