From Test-Time Scaling to Reusable Memory: Measuring Crystallization in Text-to-SQL

2026-08-07

AI Summary

This work addresses the underexplored "crystallization" problem in Text-to-SQL systems, where corrected queries are discarded at test time, obscuring the practical value of memory for future queries—such as repeated questions, novel queries, or cross-question generalization within the same database. The study formalizes this issue and introduces a controlled evaluation framework that fixes the solver while varying only the memory strategy, enabling isolated assessment of memory’s multidimensional benefits in replay, cross-question retention, and intra-database transfer. Leveraging verified corrected queries as memory content, combined with a robust validation mechanism and broad-coverage retrieval strategy, the authors systematically compare different memory formats and retrievers. On the BIRD dataset, incorporating memory improves first-attempt accuracy by 4.34 percentage points, achieving 44.4% of the theoretical upper bound attainable through on-demand repair, highlighting database-specific content as critical to memory effectiveness.

Abstract

Test-time scaling can correct difficult text-to-SQL queries, but the extra computation is normally discarded after each answer. Systems increasingly retain verified repair episodes, yet evaluations still report one end-to-end score. It cannot distinguish replay on recurring questions from help on unseen questions, or identify the responsible memory choice. We call measuring this future value the crystallization problem. Our controlled evaluation holds the single-shot solver fixed and varies one memory choice at a time. We separately measure replay, cross-question retention, and held-out same-database transfer. On BIRD, storing verified corrected queries improves held-out first-attempt accuracy by 4.34 percentage points. This gain captures 44.4% of the accuracy headroom provided by on-demand repair on the same questions. Controlled interventions identify database-specific content as the main operating ingredient. Reliable verification and broader retrieval coverage yield supported gains; richer formats and elaborate retrievers do not. Open-source code, evaluation artifacts, and reproduction instructions are available at https://github.com/ai-jiaqian/text-to-sql-memory-crystallization.

Problem

• text-to-SQL
• test-time scaling
• memory reuse
• crystallization
• evaluation

Innovation

• crystallization
• test-time scaling
• text-to-SQL
• memory reuse
• controlled evaluation