Simple-OPD: Demystifying Warm-up for On-policy Distillation

2026-08-07

AI Summary

This work investigates the pre-warming phase in on-policy distillation (OPD), which significantly impacts performance yet lacks a clear mechanistic understanding. The study reveals that the core of effective pre-warming lies in transferring inference patterns compatible with the teacher model, rather than merely relying on ground-truth labels. To this end, the authors propose Simple-OPD, a method that leverages teacher-generated chain-of-thought data to initialize student training via low-rank adaptation (LoRA) in a plug-in manner, followed by standard OPD training. Extensive experiments demonstrate that Simple-OPD consistently outperforms full-parameter supervised fine-tuning across diverse settings, exhibiting both strong effectiveness and robustness.

Abstract

On-policy distillation (OPD) trains a student on its own rollouts with token-level supervision from teacher models, but its effectiveness can depend strongly on the warm-up stage before OPD. In this paper, we demystify warm-up for OPD from both data and training perspectives. For data, we find that effective warm-up relies on teacher-compatible chain-of-thought supervision, and that even incorrect teacher rollouts can provide comparable benefits to correct ones. This suggests that warm-up primarily transfers a teacher-compatible thinking pattern rather than merely correct answers. For training, we show that low-rank adaptation (LoRA) with a near-saturation training duration better balances in-domain adaptation and out-of-distribution generalization than full-parameter SFT. Based on these findings, we propose Simple-OPD, a plug-and-play initialization method that warms up the student on teacher-generated CoT with LoRA before OPD. Experiments across diverse settings demonstrate the effectiveness and robustness of Simple-OPD.

Problem

• On-policy Distillation
• warm-up
• chain-of-thought
• teacher-compatible supervision
• student initialization

Innovation

• On-policy Distillation
• Warm-up
• Chain-of-Thought
• LoRA
• Knowledge Transfer