Problem
Research questions and friction points this paper is trying to address.
On-Policy Distillation
multilingual math reasoning
language preservation
performance gap
large language models
On-Policy Delta Distillation for Multilingual Math Reasoning
🤖 AI Summary
This work addresses the issue of target-language response drift in multilingual (English, Korean, Japanese) mathematical reasoning caused by cross-lingual transfer in large language models. To mitigate this problem, the authors propose an improved On-Policy Delta Distillation (OPD²) method that leverages the probability difference between a teacher model and its base model as the learning signal. Remarkably, OPD² enhances mathematical reasoning performance in Korean and Japanese using only English training data. Experiments based on the Qwen3 architecture demonstrate that OPD² significantly outperforms the original OPD approach, substantially narrowing the performance gap between English and Korean while alleviating imbalances in multilingual reasoning capabilities. These results validate the effectiveness of probability-difference signals for knowledge transfer across languages.
📝 Abstract
On-Policy Distillation (OPD) is emerging as a promising alternative to reinforcement learning for LLM post-training, yet its effectiveness in multilingual settings remains underexplored. We study OPD and its advanced variant, On-Policy Delta Distillation (OPD$^2$), for mathematical reasoning in English, Korean, and Japanese. OPD$^2$ improves OPD by using the probability gap between a post-trained teacher and its base model as the learning signal. Experiments with Qwen3 show that OPD$^2$ consistently outperforms the original OPD, with particularly strong improvements in Korean and Japanese, and generally narrows the English-Korean performance gap. We further find that English-only OPD can also increase performance for Korean and Japanese, but often shifts the responses toward English, highlighting the importance of multilingual data to preserving target-language responses.
Innovation
Methods, ideas, or system contributions that make the work stand out.
On-Policy Delta Distillation
multilingual math reasoning
probability gap
language preservation
LLM post-training
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
