🤖 AI Summary
This work addresses the challenge of deploying Text-to-SQL evaluation in production environments, where reliance on database schemas and reference SQL queries hinders effective monitoring. To overcome this limitation, the authors propose STEF, a schema-free evaluation framework that requires only the user question, its augmented paraphrase, and the generated SQL. STEF leverages the quality of question augmentation as a core signal, integrating semantic constraint extraction and alignment between natural language and normalized SQL representations. It introduces a composite scoring mechanism incorporating filtered alignment, semantic judgment, and confidence estimation, while supporting injection of application-specific rules via prompt templates. The framework demonstrates robustness to SQL constructs such as GROUP BY, ORDER BY, and LIMIT, enabling—for the first time—continuous, database-independent monitoring and feedback for Text-to-SQL systems in real-world deployments.
📝 Abstract
Text-to-SQL (T2SQL) evaluation in production environments poses fundamental challenges that existing benchmarks do not address. Current evaluation methodologies whether rule-based SQL matching or schema-dependent semantic parsers assume access to ground-truth queries and structured database schema, constraints that are rarely satisfied in real-world deployments. This disconnect leaves production T2SQL agents largely unevaluated beyond developer-time testing, creating silent quality degradation with no feedback mechanism for continuous improvement. We present STEF (Schema-agnostic Text-to-SQL Evaluation Framework), a production-native evaluation system that operates exclusively on natural language inputs the user question, an enriched reformulation, and the generated SQL without requiring database schema or reference queries. STEF extracts semantic specifications from both natural language and SQL representations, performs normalized feature alignment, and produces an interpretable 0 to 100 accuracy score via a composite metric that encompasses filter alignment, semantic verdict, and confidence of the evaluator. Key contributions include: enriched question quality validation as a first-class evaluation signal, configurable application-specific rule injection via prompt templating, and production-robust normalization handling GROUP BY tolerance, ORDER BY defaults, and LIMIT heuristics. Empirical results demonstrate that STEF enables continuous production monitoring and agent improvement feedback loops without schema dependency, making structured query evaluation viable at scale for the first time.