🤖 AI Summary
Existing multi-teacher knowledge distillation methods lack a theoretically grounded mechanism for adaptive weight assignment, often relying on heuristic strategies. This work proposes the first operator-agnostic axiomatic framework that enables principled adaptive weighting across three granularities—tokens, tasks, and contexts—while supporting hierarchical composition and safety constraints. By leveraging axiomatic modeling, product-structure normalization, and perturbation robustness analysis, our approach decouples theoretical guarantees from specific weighting formulations, ensuring applicability to heterogeneous models and distribution-shifted scenarios. We prove the existence (and non-uniqueness) of weighting operators satisfying the proposed axioms, establish convergence and stability guarantees for the associated optimization, and provide a formal characterization of knowledge distillation under safety constraints.
📝 Abstract
Knowledge distillation with multiple teachers is increasingly used to improve robustness, efficiency, and safety, yet existing approaches rely largely on heuristic or implementation-specific weighting schemes. This paper develops an operator-agnostic axiomatic framework for adaptive weighting in multi-teacher knowledge distillation across three complementary scales: token, task, and context. We formalize structural conditions under which adaptive weighting operators are well-defined, admit multiple non-equivalent implementations, and can be hierarchically composed via product-structure normalization. Within this framework, we establish existence and non-uniqueness of conforming operators, characterize convergence of gradient-based optimization under standard assumptions, analyze stability and perturbation robustness, and provide an abstract formulation of safety-constrained distillation. The results decouple theoretical guarantees from specific weighting formulas, enabling principled analysis of adaptive distillation methods under heterogeneity, distribution shift, and safety constraints.