model compression

This paper addresses the lack of theoretical guidance for allocating computational budget between teacher and student models in knowledge distillation. We establish, for the first time, a computational scaling law for distillation learning, quantitatively characterizing how student performance varies with teacher–student compute allocation and total budget. Methodologically, we propose a predictive distillation scaling law, derived via large-scale cross-model-size distillation experiments, computational modeling, and empirical law fitting, yielding optimal compute allocation strategies for two practical scenarios. Key contributions include: (1) uncovering how the performance crossover point—where distillation surpasses supervised pretraining—evolves with model scale; (2) identifying the critical compute threshold beyond which multi-student distillation consistently outperforms supervised training; and (3) providing a reusable, generalizable compute configuration paradigm for industrial-scale model compression, substantially reducing deployment risks in large-scale distillation.

This work addresses the challenge that conventional speech models struggle to jointly optimize performance and computational complexity during training due to their non-differentiable architectural parameters. To overcome this limitation, the authors propose a reparameterization method based on feature noise injection, which for the first time enables end-to-end differentiable, dynamic adjustment of model architecture during training. This approach facilitates simultaneous optimization of accuracy and FLOP/s without relying on post-hoc pruning or quantization. By integrating differentiable architecture search with standard SGD optimization, the method significantly reduces computational overhead while maintaining strong performance on both voice activity detection and audio anti-spoofing tasks. The implementation has been made publicly available.

To address the accuracy degradation of large language models (LLMs) under low-bit quantization, this paper proposes a post-training model expansion method that dynamically enhances 4-bit quantized LLM performance without retraining. The core innovation lies in the first systematic validation of synergistic co-design between post-training expansion and quantization. We introduce a selective, progressive parameter expansion mechanism integrated with Hadamard rotation, high-precision retention of sensitive weights, layer-wise expansion, and quantization-aware structural adaptation. Evaluated on Llama3-1B, our approach achieves full 4-bit weight and activation quantization, attaining an average zero-shot accuracy improvement of 3% over QuaRot and SpinQuant, with only a 5% parameter increase. The resulting model size is reduced by 3.8% relative to the BF16 baseline—breaking the conventional “compression implies parameter reduction” paradigm and enabling joint optimization of accuracy and efficiency.

Modern large-scale distributed training faces sharply diminishing returns in hardware scaling: as GPU counts reach thousands, communication overhead dominates performance bottlenecks, rendering conventional parallelism strategies—data, tensor, and pipeline parallelism—suboptimal. Method: Leveraging real-world LLM training workloads, this project establishes an empirical analytical framework spanning diverse model scales, hardware configurations, and parallelization strategies. It quantifies the nonlinear relationship between accelerator count and performance gain, precisely identifying critical inflection points across model, data, and compute scaling dimensions. Contribution/Results: We discover that low-communication “suboptimal” strategies become optimal at extreme scale; we empirically determine hardware selection criteria, cluster topology requirements, and optimal parallelism combinations for training billion-parameter models. Our findings provide actionable, deployment-ready optimization guidelines for trillion-parameter LLM training infrastructures.

Conventional knowledge distillation tightly couples teacher and student architectures, resulting in poor cross-architecture generalization and prohibitive retraining costs for each new student. Method: We propose the Generalized Teacher Network (GTN), the first architecture-agnostic teacher framework that models the student pool as a weight-sharing supernet and employs a capacity-aware conditional mechanism to dynamically adapt the teacher to diverse student architectures. GTN jointly trains the teacher and students in a single, distillation-aware optimization pass. Contribution/Results: GTN eliminates the need for per-student teacher training; its overhead is amortized across the student pool. Evaluated on multi-architecture student pools, GTN consistently improves accuracy by 1.2–2.8% over baseline distillation methods, significantly enhancing deployment flexibility and computational efficiency.

This work addresses the lack of systematic methodologies in model optimization, which often relies on heuristic choices and struggles to accommodate diverse deployment constraints. It formalizes model compression and acceleration as a constraint-aware multi-objective engineering decision problem, establishing a unified and actionable framework grounded in five key dimensions: data availability, latency, memory footprint, accuracy tolerance, and retraining budget. By integrating techniques such as quantization, pruning, knowledge distillation, parameter-efficient fine-tuning (PEFT), and inference optimization, the study proposes tailored optimization pipelines for four representative industrial scenarios, delivering a reproducible and quantifiable guide for technology selection.

This work addresses the challenge of deploying deep neural networks on edge and embedded devices, where limited memory and computational resources necessitate a careful balance between model compression and performance. The authors propose a two-stage compression framework: first, joint pruning and quantization drastically reduce model size; second, a Mixture-of-Experts (MoE) mechanism dynamically routes inputs among multiple lightweight submodels to recover accuracy loss while preserving efficient inference. Notably, this study presents the first unified integration of pruning, quantization, and MoE architecture for effective ensemble-based compression. Experimental results demonstrate that the proposed method substantially reduces both parameter count and FLOPs of CNNs across multiple benchmark datasets, with only negligible degradation in accuracy.

This work addresses the high computational cost and verbosity induced by long chain-of-thought (CoT) reasoning trajectories in knowledge distillation. The authors propose a post-processing compression method that significantly shortens these trajectories prior to distillation while preserving up to 96% of downstream task accuracy. By leveraging instruction-tuned models, they compress correct reasoning traces generated by Qwen3.5-397B-A17B and gpt-oss-120B, integrating efficient fine-tuning techniques such as LoRA during distillation. The compressed trajectories occupy only 8.6–21.0% of the original text length, reducing training tokens to 12–30% and accelerating training by 2.0–7.6×. Inference outputs are shortened by 3–19×, enabling smaller student models to achieve performance nearly on par with that obtained using full-length trajectories, thereby effectively balancing accuracy and efficiency.

This work addresses the lack of systematic investigation into efficient compression of Mixture-of-Experts (MoE) models during large-scale pretraining. We systematically explore the synergistic integration of structured pruning and knowledge distillation in MoE pretraining and introduce three key innovations: post-pruning reinitialization that outperforms training from scratch, a partially preserved expert merging strategy, and a multi-token prediction distillation approach. We further demonstrate that progressive pruning significantly surpasses one-shot compression. Applying our method to the Qwen3-Next-80A3B model, we compress it to 23A2B under identical training budgets while achieving downstream task performance closely matching that of the original model, thereby validating the effectiveness and competitiveness of the proposed framework.

This work addresses the high training and inference costs in Chain-of-Thought (CoT) distillation caused by verbose reasoning traces from teacher models, noting that existing compression approaches lack systematic disentanglement of key factors. The study introduces the first structured decomposition of CoT compression into three dimensions: importance criteria, reconstruction granularity, and compression budget, integrating selective pruning with generative rewriting. Comprehensive experiments across mathematical and general domains, as well as short and long CoT settings, reveal that step-level importance criteria converge toward a shared reasoning backbone; mathematical tasks are sensitive to structural perturbations, whereas general tasks benefit from aggressive rewriting; and compression during training does not necessarily reduce inference cost. The findings yield condition-aware compression guidelines that elucidate non-trivial interactions among granularity, domain specificity, and computational cost.