Harnessing Negative Signals: Reinforcement Distillation from Teacher Data for LLM Reasoning

2025-05-30

AI Summary

This work addresses the underutilization of both positive and negative reasoning trajectories in offline knowledge distillation for large language models (LLMs). We propose REINFORCEMENT DISTILLATION (REDI), a novel objective that requires neither a reference model nor explicit reward modeling. Methodologically, we introduce a two-stage framework: (1) supervised fine-tuning on teacher-generated correct reasoning traces; and (2) joint optimization over both positive and negative trajectories via REDI, explicitly modeling relative path quality. This departs from conventional paradigms relying solely on positive samples or preference-based objectives like DPO or SimPO. Evaluated on MATH-500, Qwen-REDI-1.5B achieves 83.1% pass@1—surpassing DeepSeek-R1-Distill-Qwen-1.5B (trained on 800K private samples) despite using only 131K open-source data—establishing new state-of-the-art performance for offline distillation in the 1.5B parameter regime.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recent advances in model distillation demonstrate that data from advanced reasoning models (e.g., DeepSeek-R1, OpenAI's o1) can effectively transfer complex reasoning abilities to smaller, efficient student models. However, standard practices employ rejection sampling, discarding incorrect reasoning examples -- valuable, yet often underutilized data. This paper addresses the critical question: How can both positive and negative distilled reasoning traces be effectively leveraged to maximize LLM reasoning performance in an offline setting? To this end, We propose Reinforcement Distillation (REDI), a two-stage framework. Stage 1 learns from positive traces via Supervised Fine-Tuning (SFT). Stage 2 further refines the model using both positive and negative traces through our proposed REDI objective. This novel objective is a simple, reference-free loss function that outperforms established methods like DPO and SimPO in this distillation context. Our empirical evaluations demonstrate REDI's superiority over baseline Rejection Sampling SFT or SFT combined with DPO/SimPO on mathematical reasoning tasks. Notably, the Qwen-REDI-1.5B model, post-trained on just 131k positive and negative examples from the open Open-R1 dataset, achieves an 83.1% score on MATH-500 (pass@1). Its performance matches or surpasses that of DeepSeek-R1-Distill-Qwen-1.5B (a model post-trained on 800k proprietary data) across various mathematical reasoning benchmarks, establishing a new state-of-the-art for 1.5B models post-trained offline with openly available data.

Problem

• Leveraging both positive and negative reasoning traces for LLM distillation
• Improving offline LLM reasoning performance via Reinforcement Distillation
• Maximizing small model capabilities using openly available teacher data

Innovation

• Uses positive and negative reasoning traces
• Two-stage Reinforcement Distillation framework
• Reference-free loss function outperforms DPO