RLCSD: Reinforcement Learning with Contrastive On-Policy Self-Distillation

2026-06-10

AI Summary

This work identifies and formally names a previously unrecognized issue in existing on-policy self-distillation (OPSD) methods—termed “privilege-induced style drift”—where privileged context causes the supervision signal to overemphasize stylistic tokens at the expense of task-critical content, leading to training instability or output truncation. To address this, the authors propose Reinforced Contrastive Self-Distillation (RLCSD), a novel approach that leverages contrastive learning between teacher–student output distributions under correct versus incorrect prompts to suppress task-irrelevant stylistic shifts and refocus supervision on task-relevant tokens. RLCSD is readily integrable into existing OPSD frameworks and extends naturally to cross-model on-policy distillation. Experiments on Qwen3 (1.7B/4B/8B) and Olmo-3-7B-Think demonstrate substantial improvements over GRPO and prior OPSD methods across mathematical and logical reasoning benchmarks.

Abstract

On-policy self-distillation (OPSD) provides dense, token-level supervision for reasoning models by aligning a model's own distribution with the distribution it produces under privileged context, typically a verified solution. However, we show that the learning signal drawn from this distributional gap concentrates on style tokens rather than task-bearing ones, as the hinted model tends to produce more direct, shorter outputs. We term this pathology \emph{privilege-induced style drift}, which destabilizes training or causes response length to shrink. To address this, we propose \textbf{RLCSD} (Reinforcement Learning with Contrastive on-policy Self-Distillation), which mitigates this drift by contrasting the teacher-student gap under a correct hint against that under a wrong hint, suppressing the style shift that conditioning on a hint tends to induce regardless of correctness, and yielding a signal that is more concentrated on task-bearing tokens. Experiments on Qwen3 (1.7B/4B/8B) and Olmo-3-7B-Think across mathematical and logical reasoning show that RLCSD consistently outperforms GRPO and prior OPSD methods. We further show that the contrastive principle is general: it plugs into existing OPSD methods to improve them, and its underlying insight extends to the broader cross-model on-policy distillation setting.

Problem

• on-policy self-distillation
• privilege-induced style drift
• token-level supervision
• reasoning models
• distributional alignment

Innovation

• Contrastive Self-Distillation
• On-Policy Distillation
• Privilege-Induced Style Drift
• Reinforcement Learning
• Reasoning Models