Gains and Collapse in On-Policy Distillation:A Reinforcement Learning Perspective

2026-10-02

AI Summary

This study addresses the limitations of Online Policy Distillation (OPD), which is prone to inducing verbose, repetitive text generation and model collapse, while its underlying performance gains remain poorly understood. Adopting a reinforcement learning perspective, this work introduces an implicit reward model to interpret teacher signals, revealing that OPD fundamentally amplifies existing behavioral patterns rather than extending capability boundaries. Furthermore, it identifies that preference misalignment between teacher and student models can lead to reward hacking. To mitigate these issues, the proposed approach integrates supervised fine-tuning (SFT) initialization with response masking techniques. Empirical results demonstrate that filtering out low-quality responses effectively alleviates model collapse and substantially enhances the reliability of model evaluation.

Abstract

On-policy distillation (OPD) has become an important approach to language model post-training. However, despite its performance gains, OPD can also collapse into excessively long and repetitive generation, and the mechanism underlying these divergent outcomes remains poorly understood. We explain these outcomes through a reinforcement learning perspective: the teacher implicitly rewards student behaviors, even those it rarely exhibits itself. From this perspective, our experiments show that OPD improves performance without expanding the student's capabilities. When the implicit reward model is reliable, OPD makes correct responses easier to sample. In contrast, when the preference misaligns with quality, reward hacking happens: the implicit reward model amplifies overlong, repetitive student rollouts, even though it rarely generates such text itself. Guided by this diagnosis, we find that masking unhealthy responses during training and using SFT initialization can each effectively mitigate the collapse. Together, these findings show that OPD amplifies student behaviors favored by the teacher's implicit feedback, shifting the focus from how well the teacher generates to how reliably it evaluates student rollouts. Our code is available at https://github.com/HancCui/opd_hacking.

Problem

• On-Policy Distillation
• Language Model Post-training
• Model Collapse
• Reward Hacking
• Reinforcement Learning

Innovation

• On-Policy Distillation
• Reinforcement Learning
• Reward Hacking
• Implicit Reward Model
• Language Model Post-Training