Distill Where You Fail: Recovering Learning Signals of Negative RL-Groups from Adaptive Teacher Guidance

2026-08-01

AI Summary

This work addresses the vanishing gradient issue in Group Relative Policy Optimization (GRPO) under sparse rewards and homogeneous intra-group rewards, as well as the performance degradation caused by naively combining GRPO with Output Distributional Policy Distillation (OPD). To overcome these limitations, the authors propose the Reward-Sensitive Teacher Guidance (RSTG) method, which introduces an adaptive teacher-guidance mechanism that selectively performs distillation at both the sample and token levels—specifically targeting high-entropy tokens or those exhibiting large discrepancies between student and teacher policies—and applies teacher-confidence-weighted distillation on negative zero-variance samples. Additionally, RSTG leverages teacher-generated positive trajectories for supervised fine-tuning to restore learning signals for negatively rewarded reinforcement learning groups. Experiments demonstrate that RSTG achieves significant improvements of 4.02% and 3.05% on mathematical and code-related tasks, respectively, substantially outperforming the naive GRPO+OPD baseline.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become a standard paradigm for post-training large language models (LLMs). While Group Relative Policy Optimization (GRPO) is widely adopted, it suffers from sparse reward signals and loses gradients entirely when all responses within a group receive identical rewards. On-policy distillation (OPD) offers a natural remedy by providing dense, token-level supervision from a teacher model. However, naively combining GRPO with OPD leads to degraded performance, due to three underlying causes: not all samples benefit from distillation; fitting too quickly to the teacher undermines the exploratory capacity of RL; and OPD's advantages are asymmetric, suppressing most tokens. To address these challenges, we propose RSTG (Recovering Learning Signals via Adaptive Teacher Guidance), which applies distillation selectively and precisely where it matters most. At the sample level, OPD is restricted to negative zero-variance prompts with each sample weighted by the teacher's confidence score. At the token level, distillation targets only tokens with high student entropy or large teacher-student divergence. We further augment training with SFT on correct trajectories generated by the teacher model, injecting positive gradient signals where RL yields none. Experiments demonstrate that RSTG substantially outperforms naive GRPO+OPD by +4.02% on math and +3.05% on code.

Problem

• Reinforcement Learning
• Sparse Rewards
• On-policy Distillation
• Gradient Loss
• Group Relative Policy Optimization

Innovation

• adaptive distillation
• reinforcement learning with verifiable rewards
• on-policy distillation
• teacher-guided learning
• gradient signal recovery