Joint Structural Pruning and Mixed-Precision Quantization for LLM Compression

2026-06-05

AI Summary

This work addresses the limited compression efficacy of large language models caused by error accumulation from layer-wise quantization and the suboptimal separation of pruning and quantization in existing pipelines. The paper proposes the first end-to-end joint optimization framework that simultaneously performs structured pruning and mixed-precision post-training quantization within a unified search space. By explicitly modeling global error propagation to minimize overall loss, the approach supersedes conventional layer-wise optimization strategies. Evaluated at ultra-low precisions of 1–3 bits, the method achieves substantial improvements in compression performance, reducing perplexity by up to 21% on WikiText and 85% on C4—significantly outperforming current state-of-the-art joint compression techniques.

Abstract

Recently, the efficiency of Large Language Models (LLMs) deployment has become a critical concern in practical applications. While post-training quantization (PTQ) and structural pruning are established techniques for reducing memory footprint and inference latency, most existing PTQ approaches optimize quantization errors on a per-layer basis, overlooking how errors accumulate and propagate through the network, often resulting in suboptimal solutions. Traditional pipelines also tend to apply pruning and quantization in isolation or sequentially, further compounding sub-optimality. We introduce a novel end-to-end framework that addresses these limitations in two key ways. First, we propose a novel mixed-precision PTQ strategy that directly minimizes global error propagation across the entire model, rather than isolating layer-wise errors. Building on this, we develop a novel joint optimization approach that simultaneously learns structural pruning decisions and mixed-precision quantization policies within a unified search space. Extensive experiments show that, at ultra-low precisions (1-3 bits), our quantization method reduces WikiText perplexity by up to 21% compared to state-of-the-art (SoTA) weight-activation quantization baselines. Against leading weight-only quantization methods, it achieves up to 59% and 85% lower perplexity on WikiText and C4, respectively. Compared to the SoTA joint pruning-and-quantization techniques, our proposed method delivers superior perplexity and reasoning performance at ultra-low bits.

Problem

• Large Language Models
• Post-Training Quantization
• Structural Pruning
• Error Propagation
• Model Compression

Innovation

• mixed-precision quantization
• structural pruning
• error propagation
• joint optimization
• post-training quantization