Problem
Research questions and friction points this paper is trying to address.
knowledge distillation
verbal feedback
on-policy learning
language models
post-training
Inductive Feedback for Mixed-Policy Distillation
🤖 AI Summary
This study addresses the challenge in mixed-strategy distillation where teachers' inherent preferences interfere with student models and feedback guidance is underutilized. To overcome these limitations, this work proposes an inductive feedback method that introduces a probabilistic confirmation framework to decouple teacher preferences from feedback signals for constructing target distributions. Furthermore, it designs a shared rolling estimator based on symmetric divergence minimization, which integrates importance weighting with trust region constraints to maximize feedback utilization efficiency. Extensive evaluations demonstrate that the proposed approach significantly outperforms conventional online distillation and contrastive variants across both knowledge and agent benchmarks, effectively enhancing the post-training performance of language models.
📝 Abstract
Verbal feedback can identify errors and prescribe corrections, providing rich supervision for language-model post-training even when reliable programmatic verifiers are unavailable. Such feedback, often generated by a capable model, can be used to condition the teacher in on-policy distillation, which trains the student to match the teacher's predictions on student-generated rollouts. However, this approach can transfer teacher preferences that the feedback did not motivate, while leaving much of the feedback's guidance unused. We find that both problems come from the standard on-policy distillation objective, specifically the divergence it minimizes and the distribution it uses as its target. Our proposed method addresses both limitations. First, to isolate the information conveyed by the feedback from the teacher's inherent preferences, we treat verbal feedback as evidence for or against the hypothesis that a particular token comes next at a given prefix. We then adopt a probabilistic confirmation framework which uniquely determines an ordering over the vocabulary based on the teacher's predictions before and after it receives feedback. Using a confirmation score consistent with this ordering, we construct a target distribution within a trust region of the student. Second, to learn from guidance that student rollouts can leave unused, we derive a simple shared-rollout estimator of a symmetric divergence between the student and target distributions over rollouts, reusing student and feedback-conditioned teacher rollouts in both directions through importance weighting. Empirical evaluations show that our method outperforms the common on-policy distillation recipe and a recent contrastive variant on knowledge-based and agentic benchmarks.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Mixed-Policy Distillation
Probabilistic Confirmation Framework
Verbal Feedback
Shared-Rollout Estimator
Importance Weighting
🔎 Similar Papers
No similar papers found.
