knowledge distillation

Conventional knowledge distillation tightly couples teacher and student architectures, resulting in poor cross-architecture generalization and prohibitive retraining costs for each new student. Method: We propose the Generalized Teacher Network (GTN), the first architecture-agnostic teacher framework that models the student pool as a weight-sharing supernet and employs a capacity-aware conditional mechanism to dynamically adapt the teacher to diverse student architectures. GTN jointly trains the teacher and students in a single, distillation-aware optimization pass. Contribution/Results: GTN eliminates the need for per-student teacher training; its overhead is amortized across the student pool. Evaluated on multi-architecture student pools, GTN consistently improves accuracy by 1.2–2.8% over baseline distillation methods, significantly enhancing deployment flexibility and computational efficiency.

Existing machine learning teaching frameworks overlook representational alignment between teachers and students, prioritizing only model accuracy. Method: We propose GRADE, a representation-alignment-driven pedagogical optimization framework. Leveraging controlled machine–machine and machine–human teaching experiments, we formally define and quantify the relationship between representational alignment and teaching utility, introducing the alignment-driven teaching utility curve. We further design GRADE-Match, a cross-modal teacher–student matching algorithm that optimizes representational adaptation. Contribution/Results: Experiments demonstrate that improved representational alignment significantly enhances student task accuracy—moderated by class size and representation diversity. In simulated teaching settings, GRADE-Match achieves an average 12.3% improvement in learning outcomes. GRADE establishes a novel paradigm for interpretable, optimization-aware intelligent teaching systems grounded in representational alignment principles.

In privileged imitation learning, students often fail to replicate teacher behaviors due to limited observational capabilities—stemming from a fundamental asymmetry: the teacher’s policy is not designed for the student’s partially observable setting. To address this, we propose a joint teacher-student training framework. First, we incorporate an action-divergence approximation term into the teacher’s reward function, theoretically grounded in performance bounds to mitigate imitation failure. Second, we introduce a supervised behavioral alignment step that explicitly constrains the teacher’s policy to be imitable by the student. Third, we optimize the entire system via vision-driven, end-to-end reinforcement learning. Evaluated on maze navigation, vision-guided quadrotor flight, and dexterous manipulation tasks, our approach yields substantial improvements in student policy performance, empirically validating the efficacy of enhancing teacher imitability.

This work addresses the inefficient utilization of “dark knowledge” in knowledge distillation due to teacher–student model capacity mismatch. We identify two empirical regularities in large-capacity teacher outputs: (i) low discriminability among non-ground-truth class probabilities, yet (ii) stable inter-class relative affinity relationships. Building on this, we establish the first quantitative link between teacher capacity and dark knowledge structure, proposing a novel paradigm that enhances the discriminability of non-ground-truth logits to mitigate capacity mismatch—moving beyond conventional reliance solely on teacher accuracy. Methodologically, we integrate logit softening with temperature calibration, an inter-class discrepancy enhancement module, and a multi-teacher contrastive distillation framework. Experiments on CIFAR-100 and ImageNet demonstrate significant performance gains for lightweight student networks, consistently outperforming state-of-the-art methods including FitNet and RKD. The approach proves robust across diverse teacher–student capacity configurations.

In transfer learning, teacher models’ training data are vulnerable to membership inference attacks (MIAs), yet existing work predominantly assumes attackers have black-box or white-box access to the teacher model—overlooking privacy threats when only the student model is accessible in a white-box setting. Method: This paper identifies that representational discrepancies between teacher and student hidden layers can be exploited for MIAs, and proposes the first MIA framework requiring only white-box access to the student model. It employs shadow models to calibrate student-layer representations, models teacher–student representation divergence, and infers teacher training membership via reverse inference—without requiring the teacher model itself. Contribution/Results: The method achieves high inference accuracy across four benchmark datasets and diverse transfer learning tasks. It significantly expands the threat surface of MIAs in transfer learning and provides novel empirical evidence and insights for safeguarding teacher model privacy.

This work challenges the common practice in knowledge distillation of naively matching a teacher model’s absolute feature representations, which overlooks the fact that such representations are only equivalent up to orthogonal transformations and isotropic scaling. From a geometric perspective, the paper proposes a new paradigm centered on representation equivalence classes: the student should instead learn class-invariant structures of the teacher’s representations—such as Gram matrices, centered kernel alignment (CKA), or principal subspaces—or leverage coordinate alignment for effective supervision. This framework unifies feature matching, relational distillation, and grafting approaches, revealing that logit-level matching is ultimately key to capability transfer. Experiments on Qwen2.5 and Llama-3.1 demonstrate that high CKA similarity alone is insufficient for performance recovery, while successful grafting hinges on boundary overlap in the training data coverage, thereby validating the proposed theory.

This work addresses the distribution mismatch commonly faced by large language model agents in supervised fine-tuning, where training relies on complete teacher demonstrations while testing depends on student-generated contexts. The authors formulate online policy data construction as a budget allocation problem and propose replacing lengthy or costly filtered teacher trajectories with a small number of unfiltered, short-step teacher continuations, strategically injected into student-induced critical contexts. By systematically exploring the design space of rollout policies, switching time distributions, continuation lengths, and filtering rules—and incorporating a dual-cost model accounting for both teacher inference and supervision signal retention—the method demonstrates strong empirical performance on HotpotQA, ALFWorld, and Terminal-Bench-Dev. Notably, it matches or exceeds existing critical-context filtering baselines on the first two benchmarks at lower computational cost, indicating that a few well-placed teacher steps can substantially enhance training efficiency.

This work addresses the issue of systematic bias propagation in student-teacher learning, where directly matching the teacher’s outputs can inadvertently transfer its biases to the student. To mitigate this, the authors propose “Residuals as Teachers” (RaT), a novel approach that introduces residual learning into the student-teacher framework: the teacher estimates the residual between the student’s current prediction and the target, effectively guiding the student to mimic a proximal gradient optimization process. Theoretical analysis demonstrates that RaT achieves minimax optimal convergence rates under non-asymptotic excess risk, whereas conventional soft-target matching suffers from a persistent approximation error. Empirical evaluations on both synthetic data and the ImageNette covariate shift classification benchmark confirm that RaT substantially outperforms baseline methods and effectively suppresses bias propagation.

This work addresses the challenge of transferring reasoning capabilities from large language models to smaller ones without retraining or reliance on labeled data. It introduces the “Master Key Hypothesis,” positing that model abilities are encoded along specific directions within a low-dimensional latent subspace. Building on this insight, the authors propose the UNLOCK framework, which extracts these capability directions via activation contrast, aligns subspaces across models of different scales using low-rank linear transformations, and injects the identified directions during inference to unlock latent reasoning abilities in the target small model. Experiments demonstrate substantial performance gains: for instance, Qwen1.5-7B achieves a 12.1% accuracy improvement on MATH and AGIEval Math benchmarks, while Qwen3-14B-Base even surpasses its post-trained counterpart.

This work proposes a systematic method to convert non-neural machine learning pipelines—such as those based on random forests—into neural networks, enabling unified inference and joint optimization. Leveraging knowledge distillation, the approach treats the traditional model as a “teacher” that guides the training of a neural “student” network. The framework further integrates neural architecture search with a random forest–inspired hyperparameter selection strategy to optimize the student model. Notably, this is the first effort to employ an entire non-neural machine learning pipeline as the teacher in knowledge distillation, thereby extending the scope of this technique. Experimental evaluation across 100 OpenML tasks demonstrates that the student networks consistently replicate the performance of their teacher models, confirming the feasibility and effectiveness of the proposed conversion framework.