No Pain, More Gain: Iterative Merging for Effective Multi-Teacher On-Policy Distillation

2026-09-28

AI Summary

This study addresses the susceptibility of multi-teacher online distillation to initialization effects, which hinders the effective recovery of certain domain-specific capabilities. To overcome this limitation, we propose the IM-MOPD framework, which integrates multi-teacher online distillation with parameter merging and an iterative task vector increment strategy. Starting from uniform merging, the method dynamically injects task vector increments corresponding to under-recovered domains during training, thereby transcending the constraints of conventional static coefficient search and progressively determining optimal teacher contribution ratios. Experimental evaluations across five domains demonstrate that IM-MOPD achieves significantly higher average normalized recovery rates compared to both uniform merging and SFT warm-up baselines, effectively enhancing the efficiency of multi-domain knowledge integration.

Abstract

Multi-teacher on-policy distillation (MOPD) combines independently developed domain teachers into a single student by distilling their predictions on student-generated samples. We study a setting where teachers share a reference model but undergo different post-training procedures, and find that MOPD can struggle to recover some teacher capabilities. Because distillation occurs on student-generated prefixes, the student initialization can strongly affect subsequent recovery. However, initial benchmark performance is not a reliable predictor of a good MOPD initialization. For example, merge initialization can start below SFT warm-up yet finish higher after MOPD. We further find that effective merging depends on both the relative teacher contributions and the overall merge scale, with some strong configurations lying outside the simplex of convex parameter averaging. Thus, selecting a good merge initialization requires evaluating not only its immediate performance but also the learning it enables under MOPD, making one-shot coefficient search difficult. We propose Iterative Merging for MOPD (IM-MOPD), which starts from a uniform merge and progressively adds task-vector increments for under-recovered domains during distillation. In a 5-domain setting, IM-MOPD achieves higher average normalized recovery than MOPD with either uniform merge initialization or SFT warm-up, showing that effective teacher contributions can be determined progressively during training.

Problem

• Multi-teacher on-policy distillation
• Model merging
• Knowledge distillation
• Student initialization
• Capability recovery

Innovation

• Multi-teacher on-policy distillation
• Iterative merging
• Task vectors
• Model initialization
• Knowledge distillation