Weak-to-Strong On-Policy Distillation

2026-07-28

AI Summary

This work addresses the limitation of existing on-policy distillation (OPD), which relies on teacher models stronger than the student and thus struggles when such teachers are unavailable or prohibitively expensive to train. The authors propose a weak-to-strong on-policy distillation framework that constructs a contrastive pair from two models weaker than the student, extracts a capability direction in logit space, and injects it into the student’s base model to generate a proxy teacher—proximal in distribution yet enhanced in capability. Token-level distillation is then performed using reverse KL divergence. This approach enables effective knowledge distillation from multiple weak models, allowing the student to surpass all supervisory sources in performance. Evaluated on four mathematical and three code benchmarks, the method significantly outperforms conventional OPD, with the student exceeding domain-specific teachers and selectively enhancing either reasoning structure or problem-solving procedures through distinct contrastive pairs.

Abstract

On-policy distillation (OPD), which aligns a student with the teacher's token-level distribution on the student's own rollouts, is an effective paradigm for transferring capabilities across LLMs. Prevailing approaches assume a teacher at least as capable as the student: they either distill a larger model into a smaller one, which fails at the frontier where no larger teacher exists, or consolidate multiple domain experts trained from a shared base, which requires costly training at the student's scale. We introduce Weak-to-Strong On-Policy Distillation (W2S-OPD), a simple yet effective OPD framework that improves the strong student by distilling from multiple weak models. W2S-OPD constructs a proxy teacher in logit space from a contrast pair of a positive and a negative model, both smaller than the student and cheap to obtain. Their logit difference isolates the capability direction, which is added to the student's own base model, yielding a proxy teacher that couples this direction while staying distributionally adjacent to the student. The student then distills it by minimizing the per-token reverse KL on its own rollouts. We instantiate the contrast pair as i) a post-RL expert against its pre-RL initialization, isolating the skill RL instills, ii) a larger against a smaller base model, isolating the capability from scale, and iii) a small base model with correct versus wrong hints, isolating the instance-level direction toward the solution. Across four math and three code benchmarks, W2S-OPD outperforms OPD, enables the student to surpass the domain teacher, and keeps improving the student even when every supervision source is weaker. Analysis shows different contrasts yield distinct signals: the post-RL and hint contrasts emphasize reasoning frameworks, while the scale contrast emphasizes the solving procedure. Our code will be available at https://github.com/Yu-Fangxu/W2S-OPD.

Problem

• on-policy distillation
• weak-to-strong transfer
• large language models
• capability distillation
• teacher-student paradigm

Innovation

• Weak-to-Strong Distillation
• On-Policy Distillation
• Logit-space Contrast
• Proxy Teacher
• Reverse KL Minimization