Problem
Research questions and friction points this paper is trying to address.
On-policy distillation
Reinforcement learning with verifiable rewards
Knowledge distillation
Credit assignment
Large language models
DiffGate: Difficulty-Gated Teacher Guidance for On-Policy Distillation
🤖 AI Summary
This study addresses the limitations of online distillation, which lacks trajectory-level supervision, and reinforcement learning, which suffers from sparse rewards and coarse credit assignment. To overcome these challenges, we propose a novel objective function that integrates Group Relative Policy Optimization (GRPO) with selective teacher guidance. Specifically, a verifier identifies failed trajectories, for which a teacher model provides dense update directions. Supervision signals are applied exclusively to these failed trajectories, scaled by difficulty and bounded in magnitude, effectively bridging the gap between local signals and global outcomes. When incorporated into the Reinforcement Learning with Verifiable Rewards (RLVR) framework, our approach significantly improves the pass@8 metric on code and mathematics benchmarks, demonstrating enhanced solution coverage capabilities of the model.
📝 Abstract
On-policy distillation (OPD) has emerged as a widely used paradigm for post-training large language models, reducing the train--test mismatch of conventional distillation by supervising the student on its own generated trajectories. However, existing OPD objectives remain largely token-local and outcome-agnostic, optimizing teacher--student agreement at each prefix despite reasoning quality being determined at the trajectory level. Reinforcement learning with verifiable rewards (RLVR), particularly Group Relative Policy Optimization (GRPO), provides complementary outcome-level supervision but suffers from sparse rewards and coarse credit assignment. We show that OPD and RLVR exhibit complementary blind spots: teacher signals provide dense local guidance but are weakly aligned with rollout correctness, whereas group-relative rewards capture task success but provide coarse token-level credit and vanish on all-failure groups. We introduce DiffGate, an outcome-gated objective that combines GRPO with selective, bounded teacher guidance. Teacher supervision is applied only to failed trajectories, scaled by group difficulty, and smoothly bounded to prevent extreme teacher--student discrepancies from dominating optimization. The verifier therefore determines \emph{which trajectories} receive teacher guidance, while the teacher provides dense token-level update directions within those trajectories. Across Qwen3-0.6B and Qwen3-1.7B students, DiffGate improves code avg@8 over matched GRPO by $+1.7$ and $+1.8$ points and pass@8 by $+1.6$ and $+5.7$ points, respectively. On mathematics, avg@8 remains within $0.5$ points of GRPO while pass@8 improves by $+1.1$ and $+3.9$ points. Overall, DiffGate improves pass@8 across all four model--domain settings, demonstrating improved solution coverage under our evaluation protocol.
Innovation
Methods, ideas, or system contributions that make the work stand out.
On-policy distillation
GRPO
Outcome-gated objective
Teacher guidance
Reinforcement learning with verifiable rewards
🔎 Similar Papers
No similar papers found.
