Agentic Reinforcement Learning with Observation-Calibrated Self-Distillation

2026-08-05

AI Summary

This work addresses the challenge in reinforcement learning for large language model agents under sparse trajectory rewards, where existing methods struggle to disentangle genuine future observations from spurious score perturbations introduced by replay structures, leading to inaccurate token-level update signals. To resolve this, the authors propose Observation-Calibrated Self-Distillation (OCSD), which constructs replay views that are structurally aligned but differ in the presence of future observations. This enables extraction of score residuals attributable solely to observations. OCSD modulates token-level GRPO updates at high-uncertainty steps while preserving trajectory-level optimization direction. By introducing an observation ablation mechanism, the method effectively isolates the influence of replay structure from authentic environmental feedback, yielding precise token-level supervision. Evaluated on ALFWorld, WebShop, and Search-QA, Qwen3 models of three scales consistently outperform strong baselines, with diagnostic analyses confirming that calibrated residuals better align with local environmental signals.

Abstract

Large language model agents are commonly trained through reinforcement learning with sparse trajectory-level rewards, which offer limited guidance on how strongly individual tokens should be updated. On-Policy Self-Distillation (OPSD) addresses this by re-scoring generated tokens under a privileged replay view to obtain dense, token-level supervision. However, we identify a confounding issue: the resulting support may reflect both the privileged information contained in the replay view and score shifts induced by the replay scaffold, making it difficult to attribute the support specifically to that information. This issue is especially pronounced when future environment observations serve as privileged information, since replaying them requires reconstructing an extended scaffold that itself perturbs token scores. To resolve this confounding, we propose Observation-Calibrated Self-Distillation (OCSD), which contrasts two structurally matched replay views, Full and Observation-Ablated, differing only in whether the actual future observation is present, to derive an observation residual that discounts score changes shared by the replay scaffold. OCSD then applies this residual to modulate token-level GRPO updates at high-uncertainty steps, while preserving the trajectory-level update direction. Experiments on ALFWorld, WebShop, and Search-QA across three Qwen3 model scales show that OCSD consistently outperforms strong baselines. Diagnostic analyses further confirm that the calibrated residual aligns better with local environment feedback. Our code is publicly available at https://github.com/yiy1x/OCSD.

Problem

• Reinforcement Learning
• Self-Distillation
• Privileged Information
• Token-level Supervision
• Observation Calibration

Innovation

• Observation-Calibrated Self-Distillation
• On-Policy Self-Distillation
• privileged replay
• token-level supervision
• replay scaffold