Lightning OPD 2.0: Mitigating Style Bias in Cross-Teacher On-Policy Distillation for Large Reasoning Models

2026-07-30

AI Summary

This work addresses the performance degradation in cross-teacher policy distillation caused by stylistic discrepancies—such as variations in phrasing, formatting, and reasoning pace—among teachers. To mitigate this issue, the authors propose a cross-fitted style residualization method that estimates and removes reproducible style biases at the rollout level prior to token-level distillation. This approach enables effective on-policy distillation across heterogeneous teachers without requiring the assumption of teacher consistency. Empirical results demonstrate its efficacy: the method achieves 82.4% on AIME 2024 and 63.0% on LiveCodeBench v5, substantially outperforming Lightning OPD and confirming its strong performance in both mathematical reasoning and code generation tasks.

Abstract

On-policy distillation (OPD) provides dense token-level supervision from a teacher, but its effectiveness can depend on teacher consistency, meaning that the model providing OPD supervision should also have generated the demonstrations used to train the supervised fine-tuning (SFT) reference. However, this condition is frequently violated in practice when SFT data have mixed or unknown provenance or when different models are preferred for SFT data generation and subsequent distillation. In such cross-teacher settings, even a stronger OPD teacher can yield little improvement over the SFT reference. We find that raw teacher--reference disagreement contains potentially useful context-specific teacher evidence as well as a recurring component associated with differences in wording, formatting, and reasoning cadence. We introduce Lightning OPD 2.0 with cross-fitted style residualization, which uses rollout-level cross-fitting to estimate this recurring component as an operational proxy for style-token bias and subtracts it before constructing the token-level OPD update. Across mathematical reasoning and code generation benchmarks, Lightning OPD 2.0 consistently outperforms Lightning OPD in cross-teacher settings. Starting from Klear-Reasoner-8B-SFT, Lightning OPD 2.0 reaches 82.4% on AIME 2024 and 63.0% on LiveCodeBench v5. Together, these results establish Lightning OPD 2.0 as a practical approach to cross-teacher OPD, relaxing teacher consistency as a prerequisite and allowing the SFT data generator and distillation teacher to be selected independently. Code will be released soon.

Problem

• on-policy distillation
• style bias
• cross-teacher
• large reasoning models
• supervised fine-tuning

Innovation

• On-Policy Distillation
• Style Bias Mitigation
• Cross-Teacher Distillation
• Style Residualization
• Large Reasoning Models