Problem
Research questions and friction points this paper is trying to address.
On-policy self-distillation
Recursive self-improvement
Teacher freezing
Reasoning
Recursive Self-Improvement via On-Policy Distillation for Reasoning
🤖 AI Summary
This study addresses the limitations of fixed teachers in self-distillation, which hinder iterative improvement and lead to verbose outputs. To overcome these issues, we propose a dynamic co-evolution mechanism between a privileged teacher and the student. Methodologically, we employ online policy distillation and a dynamic co-evolution algorithm to break through the static teacher bottleneck, enabling recursive self-improvement of the model. Furthermore, we introduce empirically validated concise rewriting targets for self-refined brevity learning, effectively suppressing redundant generation. Experimental results demonstrate that our approach achieves an average performance improvement of 35.62% on Qwen3-8B while reducing output length by 7.8%, thereby realizing the synergistic optimization of reasoning capability and expressive efficiency.
📝 Abstract
On-policy distillation (OPD) trains a student model by having it generate trajectories, then matching its next-token predictions with an external teacher's next-token predictions. This provides dense, token-level supervision to the student. On-policy self-distillation (OPSD) eliminates the need for the external teacher. Specifically, a second frozen copy of the student model, now given the ground truth in its context, serves as the teacher. The student model only receives the problem and learns to mimic the privileged teacher model, while the teacher remains frozen throughout training. Previous work showed that freezing the teacher is useful for training stability, but we argue that this can prevent the teacher from incorporating the improvements learned by the student during training. Our primary contribution is to address this limitation with a recursive framework built around two complementary components. First, we let the privileged teacher co-evolve with the student so that revision learned in one round can guide the next, a process we refer to as Dynamic Co-Evolution (DCE). Second, because stronger revision can also make responses too verbose and self-critical, we additionally train on shorter, verified rewrites of the model's own on-policy responses. We call this complementary objective Self-Refined Concise Learning (SRCL). Overall, our comprehensive evaluations show that DCE+SRCL outperforms OPSD across multiple model scales and four competition-level mathematics benchmarks. Specifically, on Qwen3-8B, DCE+SRCL reaches 65.97% Average@12, outperforming OPSD by 35.62 percentage points while reducing mean output length by 7.80% relative to DCE alone.
Innovation
Methods, ideas, or system contributions that make the work stand out.
On-policy self-distillation
Recursive self-improvement
Dynamic co-evolution
Self-refined concise learning
Reasoning
🔎 Similar Papers
No similar papers found.
