cross-component distillation

Existing knowledge distillation methods struggle to model the structured relationships among internal representations of teacher models, while mainstream contrastive learning objectives (e.g., InfoNCE) impose overly stringent instance discrimination constraints, disrupting relative semantic similarities among semantically proximal samples. To address these limitations, we propose Relational Representation Distillation (RRD). Its core innovations are: (1) a dual-temperature Softmax mechanism—employing a high temperature to emphasize dominant relational patterns and a low temperature to preserve secondary semantic similarities; and (2) a theoretically unified loss that bridges InfoNCE and KL divergence, enabling relative distribution alignment. Evaluated on multi-task transfer learning benchmarks, RRD significantly improves teacher–student representation alignment. Notably, on several downstream tasks, student models trained with RRD even surpass their teachers in performance—demonstrating both the effectiveness of structured relational modeling and its strong generalization capability.

This paper investigates whether the layer-selection strategy for intermediate-layer matching in knowledge distillation affects student model performance. Method: Through systematic experiments, we examine the impact of matching order—forward, backward, or random—between teacher and student layers, and propose a geometric analysis framework based on inter-layer feature cosine angles to characterize the intrinsic relationship between layer alignment and distillation efficacy. Contribution/Results: We find that matching order has negligible impact on student accuracy; hand-crafted optimal layer alignment is unnecessary. Empirical validation across multiple datasets and model pairs shows that backward or random matching achieves performance within 0.3% of the best manually aligned configuration. This significantly enhances the robustness and usability of distillation methods, challenging the conventional paradigm of explicit, architecture-specific layer alignment design.

This work addresses the limitations of existing knowledge distillation methods when integrating multiple heterogeneous strategies, which often suffer from implementation complexity, rigid combinations, and catastrophic forgetting. To overcome these challenges, the authors propose a Sequential Multi-Stage Knowledge Distillation (SMSKD) framework that applies distinct distillation techniques in successive stages. Each stage leverages a frozen reference model from the previous stage to anchor learned knowledge, while a sample-level adaptive loss weighting mechanism—based on the teacher’s true class probability (TCP)—dynamically balances knowledge retention and integration. The framework flexibly accommodates arbitrary distillation strategies and numbers of stages, consistently yielding significant accuracy improvements for student models across diverse teacher–student architectures, outperforming current baselines with negligible computational overhead.

Knowledge distillation’s impact on internal computational mechanisms remains poorly understood, particularly regarding how student models restructure, compress, or discard teacher components. Method: Using GPT2-small and DistilGPT2, we introduce an influence-weighted component alignment metric to quantify functional module alignment post-distillation. We integrate mechanistic interpretability, circuit analysis, activation tracing, and influence functions to assess alignment across multiple tasks. Contribution/Results: We find that distilled students rely on fewer—but more influential—components, challenging the “black-box equivalence” assumption. Although output behavior remains similar, internal computation undergoes significant shifts, degrading robustness and generalization. Our framework provides the first interpretable, quantitative diagnostic tool for assessing functional fidelity in model compression, advancing trustworthy and explainable knowledge distillation.

Conventional knowledge distillation tightly couples teacher and student architectures, resulting in poor cross-architecture generalization and prohibitive retraining costs for each new student. Method: We propose the Generalized Teacher Network (GTN), the first architecture-agnostic teacher framework that models the student pool as a weight-sharing supernet and employs a capacity-aware conditional mechanism to dynamically adapt the teacher to diverse student architectures. GTN jointly trains the teacher and students in a single, distillation-aware optimization pass. Contribution/Results: GTN eliminates the need for per-student teacher training; its overhead is amortized across the student pool. Evaluated on multi-architecture student pools, GTN consistently improves accuracy by 1.2–2.8% over baseline distillation methods, significantly enhancing deployment flexibility and computational efficiency.

This study addresses the challenge that structural heterogeneity in cross-modal feature representations precludes unit-level alignment in conventional knowledge distillation. To overcome this limitation, the authors propose an abstraction mechanism based on vector-quantized codebooks, which transforms teacher features into concept-level anchors to guide student learning. By introducing a task-relevance and compatibility-guided codebook selection algorithm, the proposed method circumvents the assumption of structural consistency across feature spaces, thereby enabling concept-level distillation between heterogeneous modalities without requiring direct alignment. The effectiveness of this framework is validated across multiple classification and semantic segmentation tasks, demonstrating substantial improvements in cross-modal knowledge transfer performance under heterogeneous feature conditions.

This study addresses the limitation of traditional knowledge distillation, which is confined to parameter imitation and fails to transfer the supra-parametric capabilities upon which agents rely, such as memory, tool use, and execution logic. We define agent distillation as the persistent transfer of task-solving knowledge and propose a taxonomic perspective based on knowledge retention loci, encompassing intra-model, artifact-based, execution framework, and cross-substrate dimensions. Furthermore, this work constructs a causal evaluation framework that disentangles transfer evidence from outcomes. By establishing a theoretical roadmap for multi-substrate knowledge transfer, this research lays the foundation for developing reliable, maintainable, and secure complex agent systems.

This work addresses the challenge of deploying model ensembles in resource-constrained settings, where their computational overhead is prohibitive despite performance gains. To this end, the authors propose an efficient knowledge distillation method that aligns representations between teacher and student models through layer- and token-level projection mappings into a high-dimensional embedding space. By integrating Low-Rank Adaptation (LoRA), the approach enables parameter-efficient fine-tuning with a lightweight alignment mechanism that supports parallel training. The trainable parameters are reduced to less than 1% of those in the teacher model. Evaluated on speech recognition tasks, the method achieves substantial reductions in word error rate (WER) and outperforms existing distillation techniques.

This work investigates the role of post-training knowledge distillation in building efficient small language models under data-scarce or resource-constrained settings. Through systematic analysis across varying data scales and teacher model strengths, the study demonstrates that knowledge distillation significantly outperforms supervised fine-tuning in low-data regimes, though this advantage diminishes as data volume increases. To address this limitation, the authors propose a two-stage distillation strategy that combines synthetic data with human-annotated examples, consistently enhancing student model performance on domain-specific tasks. Experiments on the Tulu 3 dataset further reveal that stronger instruction-tuned teacher models can restore distillation’s superiority even at higher data scales, offering a practical and effective model compression approach for resource-limited environments.

This work addresses the limitations of existing knowledge distillation approaches, which struggle to capture the complex and dynamically evolving knowledge of teacher models in later training stages, thereby constraining student performance and suffering from suboptimal distilled data quality. To overcome these challenges, the authors propose a staged teacher modeling framework coupled with a shortcut trajectory construction strategy. By leveraging a stage-aware mechanism, the method precisely captures the temporal evolution of teacher knowledge across different training phases, effectively mitigating optimization instability and inter-stage knowledge gaps. The approach significantly enhances the stability, representational capacity, and efficiency of multimodal image–text distillation, achieving state-of-the-art results on Flickr30k and COCO benchmarks—with up to a 13.5% improvement on Flickr30k (averaging 9.53%)—while simultaneously reducing storage overhead.