🤖 AI Summary
This study systematically investigates the feedback-to-update mechanism in on-policy distillation (OPD), where data are generated by the current policy. Framing OPD as a feedback-to-update problem, the work introduces a formula-driven categorization framework that unifies two major update pathways: distributional loss and policy gradient–style log-ratio updates. It further incorporates novel perspectives from temporal credit assignment and temporal vocabulary routing. By leveraging techniques such as KL divergence orientation, generalized advantage estimation (GAE), and counterfactual routing, the analysis reveals that OPD performance critically depends on state compatibility and support set construction. The paper establishes a comprehensive analytical framework for OPD, derives explicit bias bounds, and proposes new methods—GAE-OPD and CR-OPD—to enhance training stability, alongside actionable diagnostic tools and a practical implementation checklist.
📝 Abstract
On-policy distillation (OPD) trains an LLM on states induced by the current or recent student policy: the student generates complete or partial rollouts, a teacher or self-teacher scores the resulting tokens under their generated contexts, and dense log-probability, logit, or distributional signals are converted into post-training updates. This survey studies OPD as a feedback-to-update problem rather than a single loss family. We develop a formula-driven taxonomy from two routes -- direct distributional losses and policy-gradient-style log-ratio updates -- and use it to organize core methods, verifier- or outcome-guided hybrids, industrial reports, framework implementations, failure modes, and stabilization recipes under explicit evidence boundaries. The taxonomy shows that OPD effectiveness depends not only on KL direction or teacher access, but also on state compatibility, support construction, temporal credit, vocabulary-level probability routing, gates and weights, and regularization. We further separate two mechanisms often conflated in sampled-token OPD stability discussions. Temporal credit asks how teacher-student log-ratio returns should weight sampled actions across a rollout; vocabulary routing asks where probability mass should move when negative feedback suppresses a sampled token. This distinction yields bias boundaries for immediate, return-to-go, discounted, and baseline-corrected estimators, motivates GAE-OPD as a value-based hypothesis for log-ratio returns, and motivates Counterfactual Routed OPD (CR-OPD) for routing probability mass toward teacher-supported, student-reachable alternatives. We close by mapping actionability diagnostics, failure mechanisms, case studies, open problems, and a reporting checklist onto the same feedback-to-update variables.