The Stochastic Shift: A New Evaluation Paradigm for Text-to-SQL with AI Operators

2026-09-17

AI Summary

本文针对AI增强SQL查询的正确性评估问题，提出了一种多层次评估框架，该方法将确定性数据库逻辑与灵活的AI语义解耦，提高了评估准确性。

Abstract

SQL has been augmented with AI operators, enabling modern data analytics platforms to derive insights from both structured and unstructured data. We observe that while current Text-to-SQL systems can successfully generate these AI-augmented queries, reliably evaluating their correctness remains a critical open challenge. Current metrics, which rely on exact query results and deterministic execution, systematically fail against the flexible, non-deterministic outputs of AI operators. In this paper, we formalize these unique evaluation failure modes and introduce a Multilayered Evaluation Framework that decouples deterministic database logic from flexible AI semantics. We test our approach across both industry (BigQuery) and academic (ThalamusDB) systems. We demonstrate that traditional Execution Accuracy severely penalizes valid queries, achieving as low as a 25% detection rate for correct translations. Furthermore, even a state-of-the-art LLM-based autorater falsely rejects 32% of accurate queries due to the complexity of judging both relational and AI components simultaneously. By validating the standard relational logic and the AI operations separately, our framework achieves state-of-the-art overall accuracy across both platforms (up to 97.2%), proposing a reliable standard for benchmarking AI-powered SQL generators.

Problem

• Text-to-SQL
• AI Operators
• Evaluation
• Non-deterministic Outputs
• Query Correctness

Innovation

• Multilayered Evaluation Framework
• AI Operators
• Text-to-SQL