Problem
Research questions and friction points this paper is trying to address.
On-policy distillation
trajectory alignment
teacher reliability
token-level supervision
preference consistency
SG-OPD: Sign-Gated On-Policy Distillation via Sign-Consistency Gating and Phased Teacher Sampling
🤖 AI Summary
This work addresses two unrealistic assumptions commonly made in on-policy distillation methods: strict alignment between student and teacher trajectories and uniformly reliable teacher preferences across all tokens. To overcome these limitations, the authors propose a trust mechanism based on a binary verifier that, during cold-start training, blends teacher trajectories endorsed by the verifier and dynamically adjusts the update strategy according to the directional consistency between the teacher and the verifier—extrapolating when consistent and interpolating when inconsistent. This is achieved through a sign-consistency gating mechanism and a staged teacher sampling strategy, which together mitigate performance degradation caused by unreliable teacher signals. Experiments demonstrate that the proposed approach significantly outperforms standard on-policy distillation on competitive mathematical reasoning benchmarks, yielding improvements of 1.98 and 7.50 percentage points in token-level and problem-level accuracy, respectively.
📝 Abstract
On-policy distillation (OPD) trains a student on its own trajectories with dense per-token supervision from a stronger teacher, and often outperforms off-policy distillation and standard reinforcement learning. However, we find that its effectiveness implicitly relies on two assumptions that frequently break in practice: trajectory-level alignment between the student and the teacher, and uniform token-level reliability of the teacher's preferences. We therefore propose Sign-Gated On-Policy Distillation (SG-OPD), which uses a binary verifier as a trust signal for the teacher at two complementary granularities: phased teacher sampling mixes in verifier-endorsed teacher rollouts at cold-start, and a sign-consistency gate extrapolates the distillation update on tokens where the teacher agrees with the verifier-correct direction and interpolates it where it disagrees. Experiments on competition-level mathematical reasoning benchmarks show that SG-OPD consistently outperforms standard OPD, with average gains of 1.98 and 7.50 at the per-sample and per-question levels, respectively.
Innovation
Methods, ideas, or system contributions that make the work stand out.
On-Policy Distillation
Sign-Consistency Gating
Phased Teacher Sampling
Trust Verification
Mathematical Reasoning
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
