SERL-SQL: Selective Hindsight Distillation for Text-to-SQL Reinforcement Agentic Learning

2026-08-01

AI Summary

This work addresses the challenge of coarse-grained credit assignment in existing reinforcement learning approaches for Text-to-SQL, which rely solely on trajectory-level execution accuracy rewards. To enable fine-grained credit assignment, the authors propose SERL-SQL, a framework that samples interaction trajectories via policy rollouts and introduces a dedicated teacher model trained to re-score actions using execution feedback. By converting the likelihood discrepancy between student and teacher into masking weights, SERL-SQL selectively reweights GRPO advantages only on SQL and tool-action tokens, thereby localizing credit assignment. The method innovatively integrates selective post-hoc distillation with lightweight execution feedback and employs an execution-reward-guided candidate selection strategy to approximate the Best-of-N performance upper bound. Experiments show that SERL-SQL achieves 76.56% and 89.92% execution accuracy on BIRD-Dev and Spider-Test, respectively, significantly outperforming consensus-based selection methods and approaching oracle-level performance.

Abstract

Recent Text-to-SQL systems increasingly rely on multi-turn interaction, execution feedback, and reinforcement learning. However, most existing methods use execution correctness only as a trajectory-level reward, which provides limited guidance for identifying the SQL decisions responsible for success or failure. We propose SERL-SQL, a selective execution-grounded reinforcement learning framework for multi-turn Text-to-SQL agents. SERL-SQL samples on-policy SQL interaction trajectories and uses a training-only teacher to re-score student actions with execution feedback. The resulting teacher--student likelihood gap is converted into bounded, masked weights that reweight GRPO advantages only on SQL and tool-action tokens. In this way, task rewards preserve the optimization direction, while execution hindsight provides localized credit assignment. Experiments on BIRD, Spider, and cross-domain benchmarks show that SERL-SQL achieves competitive performance, reaching 76.56% execution accuracy on BIRD-Dev and 89.92% on Spider-Test. Moreover, our reward-based selection strategy closely approaches the oracle Best-of-N upper bound and consistently outperforms consistency-based selection, showing that SERL-SQL produces high-quality candidates that can be reliably identified by lightweight execution-grounded rewards. Our code will be released at https://github.com/Ffunkytao/SERL-SQL.

Problem

• Text-to-SQL
• reinforcement learning
• credit assignment
• execution feedback
• multi-turn interaction

Innovation

• Selective Hindsight Distillation
• Execution-Grounded Reinforcement Learning
• Credit Assignment
• Text-to-SQL
• GRPO