Smaller Models, Better Rejects: Preference Distillation Scaling

2026-09-30

AI Summary

This study addresses the prohibitive computational costs of preference distillation, which typically relies on large models to generate negative samples. Challenging the prevailing assumption that rejected responses require models of equal or greater scale, this work proposes a hybrid strategy leveraging small frozen models for rejection sampling, coupled with a filtering mechanism based on reference policy likelihoods. By integrating Direct Preference Optimization (DPO) with sequence-level knowledge distillation and theoretically optimizing the approach through finite-horizon utility bound analysis of linearized feature models, the framework achieves rigorous theoretical grounding. Empirical evaluations on code generation and mathematical reasoning tasks demonstrate that student models trained via this strategy outperform conventional self-generated sample baselines while substantially reducing inference overhead. Ultimately, this research establishes a novel paradigm for efficient preference alignment in language models.

Abstract

Preference distillation typically treats a teacher response as preferred and the student's own response as rejected. This assumes that self-generated failures are the most informative negatives and that rejects must come from a model at least as large as the student, making generation costly at scale. We find neither assumption holds: across students from 7B to 72B, smaller frozen models generate rejects with less inference compute yet train stronger students than self-generated rejects, before and after sequence-level knowledge distillation, on code generation and mathematical reasoning. To explain this result, we derive a finite-horizon utility bound for Direct Preference Optimization in a linearized feature model. The bound characterizes favorable reject distributions and motivates three interventions. First, mixing rejects from smaller and student-scale models improves performance as the smaller model's share increases. Second, reassigning rejects to other prompts and shuffling their code tokens still outperform length-matched gibberish, showing that task structure contributes to reject utility. Third, selecting candidates with lower likelihood under the reference policy improves net transfer when higher-likelihood candidates provide less useful contrast. Lower-likelihood selections outperform higher-likelihood ones for every source. These results suggest that effective rejects preserve task structure while limiting coupling to the reference policy, and that smaller frozen models can provide them at low cost.

Problem

• Preference Distillation
• Direct Preference Optimization
• Negative Samples
• Model Scaling
• Knowledge Distillation

Innovation

• Preference Distillation
• Direct Preference Optimization
• Reject Sampling
• Knowledge Distillation
• Smaller Models