Problem
Research questions and friction points this paper is trying to address.
On-Policy Distillation
spurious signals
input-groundedness
token-level supervision
language model priors
When Teachers Mislead: Spurious-Signal-Aware On-Policy Distillation
🤖 AI Summary
This work addresses a critical limitation in policy distillation: teacher models often provide token-level supervision signals that rely on linguistic priors or stereotyped templates rather than task-relevant input, leading student models to learn spurious correlations. To mitigate this issue, the authors propose SA-OPD, a novel framework that introduces input dependency as a key criterion for filtering distillation signals. SA-OPD employs lightweight proxy metrics to evaluate, for each token, both its dependency on the input and its distillation bias, selectively filtering out misleading updates with low input dependency and high bias. The method is applicable to both large language models and vision-language models, consistently outperforming standard OPD and existing selective distillation approaches across multiple benchmarks, thereby demonstrating the effectiveness and generality of input-dependency-guided signal selection.
📝 Abstract
On-Policy distillation (OPD) transfers teacher capabilities by supervising student-sampled trajectories with dense token-level teacher signals. Recent selective OPD methods improve this process by prioritizing signals that are confident, informative, or learnable. However, the assumptions overlook a fundamental failure mode of language models: their token-level judgments can be driven by input-agnostic language priors, formatting conventions, or stereotyped reasoning templates rather than task-specific evidence. We refer to such optimization-relevant but weakly input-grounded supervision as spurious signals in OPD, which may produce large gradients while contributing little task-improving direction. To mitigate this issue, we propose SA-OPD, a Spurious-Signal-Aware On-Policy Distillation framework that identifies and filters misleading token-level supervision based on input-groundedness and optimization impact. SA-OPD introduces a lightweight input-groundedness proxy estimating whether a token-level distillation signal truly depends on the input. It then filters only tokens that simultaneously exhibit low input-groundedness and extreme distillation divergence, thereby removing high-impact spurious updates and achieving fine-grained OPD optimization. Extensive experiments on both large language model (LLM) and vision-language model (VLM) settings demonstrate that SA-OPD consistently outperforms Vanilla OPD and competitive selective methods. These results establish input-groundedness as a key dimension for OPD supervision selection and offer a simple, effective strategy for mitigating spurious updates.
Innovation
Methods, ideas, or system contributions that make the work stand out.
On-Policy Distillation
Spurious Signals
Input-Groundedness
Token-Level Supervision
Selective Distillation
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
