Distill What You Trust: Reliability-Aware Multi-Teacher On-Policy Distillation

2026-09-20

AI Summary

本文提出TrustMOPD方法，通过无标签、基于令牌的监督分配解决多教师在线策略蒸馏中固定教师选择的问题，提高生成任务的性能。

Abstract

Multi-teacher on-policy distillation allows a student to learn from complementary specialists on its own trajectories. Domain-routed approaches, however, select one teacher per example and keep it fixed throughout the response. This design both depends on labels that mixed training corpora often lack and cannot adapt teacher selection when the expertise required changes within a trajectory. We propose \textbf{TrustMOPD}, which replaces example-level teacher selection with label-free, token-level supervision allocation. At each student-generated prefix, TrustMOPD uses each specialist's RL-induced displacement from a shared pre-RL reference as a proxy for local reliability, calibrates these scores across teachers, and constructs a weighted distillation target. Across mathematics, code, and instruction following, TrustMOPD outperforms the strongest label-free baseline, increasing the recovery ratio from $54.4\%$ to $91.5\%$ on \textsc{SingleCap} and from $54.5\%$ to $98.0\%$ on \textsc{MultiCap}, while approaching label-based MOPD on \textsc{SingleCap}. Randomizing token-level weights independently of the student-generated prefix performs no better than uniform weighting, supporting the importance of conditioning supervision on the evolving generation context.

Problem

• multi-teacher on-policy distillation
• domain-routed approaches
• label-free
• specialist selection

Innovation

• multi-teacher on-policy distillation
• token-level supervision allocation
• reliability-aware
• label-free
• dynamic teacher selection