🤖 AI Summary
Existing natural language interfaces to databases lack systematic evaluation frameworks and design theories. This work proposes QUEST, a novel framework that integrates the general-purpose FAR operation schema—Filter, Aggregate, Return—with the W5H semantic dimensions (Who, What, Where, When, Why, How) to enable structured analysis and evaluation of text-to-SQL query semantics. Through semantic annotation and structural parsing, the study validates the universality of the FAR schema across five cross-domain datasets comprising 120,464 queries. The analysis further reveals significant inter-domain disparities in semantic distributions: for instance, medical queries predominantly focus on WHEN and WHO, while WHY and HOW are nearly absent, underscoring a critical challenge for machines in performing deep reasoning over structured data.
📝 Abstract
Natural language interfaces to databases have gained popularity, yet the theoretical foundations for evaluating and designing these systems remain underdeveloped. We present QUEST (Query Understanding Evaluation through Semantic Translation), a framework resting on two independently motivated components: the FAR structural invariant, which holds that every well-formed query reduces to Filter, Aggregate, and Return operations; and the W5H dimensional framework, which holds that all filtering criteria map to six semantic dimensions (Who, What, Where, When, Why, and How). Validated across five text-to-SQL datasets (n = 120,464), FAR conformance is universal across all domains and schema types, while W5H dimensional profiles vary substantially. Healthcare queries are strongly concentrated in temporal (WHEN: 80.4%) and person-centric (WHO: 73.0%) dimensions far exceeding general-domain benchmarks, and causal (WHY) and mechanistic (HOW) reasoning are near-zero everywhere, with apparent HOW exceptions reflecting quantitative aggregation rather than genuine procedural reasoning. These results identify a frontier that must be crossed for genuine machine reasoning over structured data.