Contrastive On-Policy Distillation

2026-07-21

AI Summary

Existing on-policy distillation methods only match token-level distributions, which is insufficient to guide student models toward more efficient reasoning strategies. This work proposes COPD, a contrastive on-policy distillation framework that introduces contrastive signals into this paradigm for the first time. COPD leverages a frozen teacher model to score tokens generated under lightweight and heavyweight reasoning instructions, constructing token-level advantage signals via log-probability differences to explicitly optimize the student model’s preference over reasoning paths. The framework seamlessly extends to a teacher-free self-distillation setting (OPSD). Evaluated across nine multimodal benchmarks, COPD significantly reduces reasoning length without compromising performance, thereby enhancing inference efficiency across diverse tasks and model scales.

Abstract

On-policy Distillation (OPD) supervises a student model on trajectories sampled from its own policy by minimizing the divergence between the output distributions of the teacher and student at each token position, thereby providing dense token-level supervision. Although existing OPD methods have demonstrated strong performance in improving the reasoning ability of student models, their objectives fundamentally rely on token-level distribution matching. Consequently, they lack an explicit signal for comparing a token's relative compatibility across reasoning modes and thus do not directly model preferences between these modes. To address this limitation, we propose COPD, a contrastive OPD framework. Specifically, for each token generated by the student model, a frozen teacher model scores the same student state under two contrasting instructions that elicit light and heavy reasoning. The difference between the resulting log probabilities serves as a token-level advantage signal to guide the OPD update. Rather than merely imitating a single teacher distribution, COPD directly encourages the student model to learn more concise and efficient reasoning strategies. We conduct experiments on nine multimodal benchmarks covering both reasoning and understanding tasks. The results show that COPD substantially reduces reasoning length without compromising model performance and consistently improves efficiency across different tasks and model scales. Furthermore, the contrastive formulation can be seamlessly integrated into the On-policy Self-distillation (OPSD) framework, where self-contrastive supervision is constructed without an additional teacher model, thereby enabling the model to distill itself toward lightweight reasoning.

Problem

• On-policy Distillation
• reasoning modes
• token-level supervision
• preference modeling
• distribution matching

Innovation

• Contrastive Learning
• On-Policy Distillation
• Reasoning Efficiency
• Token-level Advantage
• Self-distillation