Layer-wise Curriculum Learning for Efficient LLM Compression

2026-09-16

AI Summary

该论文提出了一种分层课程学习方法来有效压缩大型语言模型，通过从简单到复杂的任务逐步转移知识，并采用特征缓存和多线程策略提高计算效率。

Abstract

In this paper, we introduce layer-wise curriculum learning for efficient LLM compression. The proposed method facilitates the knowledge transfer from the teacher model to the student model, utilizing a curriculum learning approach that begins with easier optimization tasks and progressively tackles harder ones. In order to adopt the layer-wise learning in LLM compression, we partition the whole model into multiple segments consisting of layers, thereby enabling more computationally efficient knowledge transfer for LLMs. Based on our theoretical analysis of cumulative error phenomenon, layer-wise curriculum learning accelerates convergence while stabilizing the knowledge transfer process. In addition, we present a feature caching method with a multi-threading strategy to efficiently address feature misalignment across layers, maximizing GPU utilization. Consequently, our method exhibits advanced model compression performance, as well as high computational efficiency in terms of minimized memory usage and short training hours. Experiments on multiple datasets show that the proposed method achieves state-of-the-art performance while reducing GPU memory usage and training hours by more than 50\% on BERT and GPT-2. Moreover, it outperforms the other pruning methods on LLaMA-family and Qwen models under the same training hours, with a lower GPU memory footprint.

Problem

• LLM Compression
• Knowledge Transfer
• Curriculum Learning
• Efficiency

Innovation

• layer-wise curriculum learning
• efficient LLM compression
• feature caching
• multi-threading strategy