PADD: Path-Aligned Decompression Distillation for Non-Router Teacher to Guide MoE Student Learning

2026-06-08

AI Summary

This work addresses the challenge of efficiently distilling knowledge from a dense teacher model into a mixture-of-experts (MoE) student under a fixed computational budget while simultaneously optimizing the routing strategy. The authors propose Path-Aligned Decompressed Distillation (PADD), a two-stage framework that first enhances expert diversity through neuron clustering and expert warm-up, then performs joint training via online adaptive distillation, path-wise refinement, and reward-augmented load balancing. PADD achieves the first effective dense-to-MoE knowledge transfer without explicit routing supervision, significantly outperforming strong baselines on mathematical reasoning benchmarks. The resulting MoE student matches or even surpasses the teacher’s performance at equivalent inference cost while exhibiting stable routing behavior.

Abstract

As large language models (LLMs) continue to scale, it becomes increasingly challenging to grow model capacity under fixed computation budgets. We propose Path-Aligned Decompression Distillation (PADD), a framework for distilling knowledge from dense teachers without explicit routing into mixture-of-experts (MoE) students while learning high-quality routing policies. PADD organizes knowledge distillation into four stages in two phases: an initialization phase (Stage I) that builds diverse functionality in the student's experts through teacher neuron clustering and student-expert warmup, and a training phase (Stages II--IV) that integrates online adaptive distillation, path-refined policy optimization, and reward-augmented load balancing in a single training pipeline. Experiments on mathematical reasoning benchmarks demonstrate that PADD yields substantial gains over strong baselines at the same inference cost and that the MoE student can match or surpass its dense teacher. They also demonstrate effective teacher-to-student knowledge distillation and stable routing behavior.

Problem

• knowledge distillation
• mixture-of-experts
• routing policy
• dense teacher
• model compression

Innovation

• Path-Aligned Decompression Distillation
• Mixture-of-Experts
• Knowledge Distillation
• Routing Policy Optimization
• Load Balancing