PCSD: Persistent Consistency for Self-Distillation in Agentic Reinforcement Learning

2026-08-03

AI Summary

This work addresses the challenge of obtaining fine-grained supervision signals for large language model agents in sparse-reward reinforcement learning settings. The authors propose Persistent Consistency Self-Distillation (PCSD), which introduces, for the first time, a token-level dynamic weighting mechanism grounded in the local persistence of teacher signals. PCSD generates continuous weights through trend-aware modulation and Sigmoid gating, and employs an adaptive window with exponential decay aggregation to capture the stability of teacher support. Integrated with GRPO, this approach jointly optimizes dense self-distillation guidance and sparse environmental feedback. On the ALFWorld benchmark, PCSD substantially outperforms GRPO by +15.6/+13.3 points and SDAR by +6.2/+5.5 points, and achieves a 15.8-point lead over GRPO on the unseen test set; it also demonstrates competitive performance on WebShop.

Abstract

Large language model agents have shown strong potential in complex interactive tasks, yet their reinforcement learning (RL) is often hindered by sparse rewards, as a long multi-turn trajectory may receive only a single outcome-level signal. On-policy self-distillation (OPSD) provides dense token-level supervision from a privileged teacher, but the teacher may not be reliable at every position. Existing methods commonly rely on isolated token-level discrepancies, which can be sensitive to noise, or assign a shared step-level weight that may overlook positional variation. We propose Persistent Consistency Self-Distillation (PCSD), which derives token-level distillation weights from the local persistence of teacher-favoring signals. PCSD combines adaptive windows with exponentially decayed aggregation to capture persistent relative teacher support, applies trend-aware modulation to attenuate locally declining support, and produces continuous weights through sigmoid gating. The resulting objective is jointly optimized with GRPO, combining dense teacher guidance with sparse environmental feedback. Without inference-time skills, PCSD achieves the best ALFWorld Overall results among all baselines on both backbones, exceeding GRPO by 15.6 and 13.3 points and SDAR by 6.2 and 5.5 points, while remaining competitive on WebShop and gaining 15.8 points over GRPO on unseen ALFWorld split.

Problem

• sparse rewards
• self-distillation
• token-level supervision
• teacher reliability
• reinforcement learning

Innovation

• Persistent Consistency
• Self-Distillation
• Token-level Weighting
• Adaptive Window Aggregation
• Trend-aware Modulation