DASH: Divergence-Adaptive Supervision Horizons for On-Policy Self-Distillation of Reasoning Models

2026-08-06

AI Summary

This work addresses the limitations of existing On-Policy Self-Distillation (OPSD) methods, which struggle to effectively leverage sparse reward signals and lack the ability to dynamically adjust supervision weights according to the temporal evolution of local discrepancies during generation. To overcome these issues, we propose Difference-Adaptive Supervision Horizon (DASH), a novel mechanism that operates within the OPSD framework by comparing local distillation signals against sequence-level averages to dynamically modulate propagation gates in multi-step backward aggregation. This enables temporally aware, token-level adaptive assignment of supervision weights, incorporating the temporal dynamics of local differences into weight scheduling for the first time—without requiring additional forward computations. Experiments demonstrate that DASH consistently outperforms standard OPSD across three mathematical reasoning benchmarks and three model scales, significantly enhancing reasoning performance with no extra computational overhead.

Abstract

Reinforcement learning with verifiable rewards (RLVR) improves the reasoning capabilities of large language models using automatically verifiable outcome signals, but these signals are typically sparse and at the sequence-level. On-policy self-distillation (OPSD) mitigates this sparsity by querying a privileged teacher at student-visited prefixes and providing dense token-level distributional supervision. Although this dense supervision alleviates signal sparsity, we find that standard OPSD still underexploits the temporal structure of the rollout. It assigns every local divergence the same coefficient, regardless of its position or the divergence sequence in which it occurs. In on-policy autoregressive generation, the same divergence magnitude can follow different discrepancy histories, reflecting different evolutions of the mismatch between the teacher and student. Since the local scalar alone cannot distinguish these temporal contexts, standard OPSD cannot adapt its token-level weights to the realized discrepancy sequence. To address this limitation, we propose Divergence-Adaptive Supervision Horizons (DASH). DASH maps the gap between each local distillation signal and the sequence-level mean to an adaptive propagation gate and then uses these gates to control backward multi-step aggregation. By doing so, DASH adjusts token-level supervision weights according to how local divergences evolve during generation. Experiments on three mathematical reasoning benchmarks across three model scales show that DASH improves over our matched vanilla OPSD reruns on every benchmark at all three scales. DASH reuses the teacher and student distributions that OPSD already computes, so the gains require no additional teacher or student forward pass. Code: https://github.com/DBtxy/DASH-OPSD

Problem

• on-policy self-distillation
• temporal structure
• token-level supervision
• divergence adaptation
• reasoning models

Innovation

• on-policy self-distillation
• divergence-adaptive supervision
• temporal structure
• token-level supervision
• reasoning models