Privileged, but Biased: How PI-Conditioned Teachers Break Self-Distillation

2026-08-05

AI Summary

This study investigates whether self-distillation (SD), when employed as a standalone optimization objective, genuinely enhances model reasoning capabilities, with particular attention to its observed tendency to degrade validation accuracy on complex tasks. By extending SD to challenging domains—including question answering, mathematical reasoning, code generation, and multi-turn tool use—we find that token-level supervision from the teacher model, conditioned on privileged information (PI), often misguides the student away from correctness. To explain this phenomenon, we introduce a causal chain of PI-induced bias and propose a PI Bias Score to quantify teacher deviation, revealing a fundamental misalignment between SD’s loss signal and task success. Extensive experiments across diverse tasks, model scales, and PI formulations demonstrate that while SD consistently reduces training loss, it systematically harms validation performance, underscoring its inadequacy as a sole objective for improving reasoning ability.

Abstract

Self-distillation (SD) has emerged as a compute-efficient alternative to reinforcement learning with verifiable rewards: a self-teacher, conditioned on privileged information (PI) about the answer such as a reference solution, supplies dense per-token supervision to a student that never sees it. Reported gains, however, come almost exclusively from narrow, low-difficulty settings, leaving open a basic question: as a lone objective, with no reward term, does SD teach anything? We reproduce SDPO's reported gains in its easy setting, then apply the identical setup to difficult tasks and find that it does not. Across question answering, mathematics, coding, and multi-turn agentic tool use, across reasoning modes, model sizes, and forms of PI, and under both the SDPO and OPSD recipes, the per-token loss falls steadily while validation accuracy does not improve and typically degrades. We explain this failure through a single causal chain from the loss to the model it produces. The chain begins with PI bias: having seen one particular reference solution, the teacher's per-token target is pulled toward that trajectory rather than toward correctness in general, an effect we quantify with a PI Bias Score. Trained to match this target everywhere, the student's objective becomes nearly blind to whether a rollout is correct, and the loss it assigns falls mostly on low-information tokens like stopwords, punctuation, uncertainty markers, rather than those that determine the answer; within correct rollouts the exploratory tokens incur the highest divergence, so it penalizes the hesitation that reasoning requires. The result is a flatter, less decisive student that is no better at reasoning: as a lone objective, SD optimizes a signal decoupled from task success.

Problem

• Self-Distillation
• Privileged Information
• Bias
• Reasoning
• Task Success

Innovation

• self-distillation
• privileged information bias
• PI Bias Score
• reasoning degradation
• token-level supervision