DAPD: Dual-Anchored Policy Distillation

2026-08-03

AI Summary

This work addresses the issue of "privileged hallucination" in online policy distillation for language model post-training, where students learn behaviors that cannot be replicated at inference due to information asymmetry between teacher and student. The study formally identifies the root cause of this problem and introduces a Dual-Anchored Policy Distillation framework. This approach employs Dual-Path Anchoring (DPA) to align behavioral trajectories with and without privileged information, and Dual-Source Anchoring (DSA) to reduce reliance on privileged signals—thereby preserving effective supervision while preventing hallucination transfer. Evaluated on Qwen3-4B and Qwen3-32B models, the method achieves average performance gains of +2.69 and +2.78, respectively, significantly mitigating privileged hallucination.

Abstract

On-policy (self) distillation (OPSD) is increasingly adopted for language-model post-training. It strengthens the teacher with privileged information but can induce a privilege illusion: the student learns privilege-dependent behavior it cannot reproduce from its inference-time context, yet behaves as if the training-time privileged information remained available, ultimately degrading performance. In this paper, we identify information asymmetry between the privileged teacher and the student at inference as the root cause of this failure in OPSD. To resolve this asymmetry, we propose Dual-Anchored Policy Distillation (DAPD), a unified framework with two levels of anchoring. Dual-Path Anchoring (DPA) introduces a self-conditioned bridge and aligns reference and rollout behavior along two matched-information paths, preventing privilege-dependent behavior from being transferred to the inference-time student. Dual-Source Anchoring (DSA) applies these paths in both reference-to-rollout and rollout-to-reference directions, reducing reliance on privileged reference guidance while preserving correctness supervision. Extensive experiments show that DAPD significantly alleviates privilege illusion, outperforming OPSD on Qwen3-4B by +2.00 points on average across tasks. Notably, its gains persist across scales, reaching +2.69 at 4B and +2.78 at 32B.

Problem

• privilege illusion
• information asymmetry
• policy distillation
• on-policy distillation
• language model post-training

Innovation

• Policy Distillation
• Privilege Illusion
• Information Asymmetry
• Dual-Anchored Framework
• Language Model Post-Training