Unified Static-Dynamic Pruning for Efficient LLM Inference

2026-07-24

AI Summary

This work addresses the computational and memory bottlenecks in autoregressive inference of large language models, where existing pruning methods struggle to balance efficiency and adaptability. The authors propose SPDP, a novel framework that unifies static and dynamic pruning for the first time, introducing the Tiled-CBC sparse storage format alongside optimized GPU kernels: a CUDA Core–based spMspV kernel integrated with HAD-SMBD and a Tensor Core–optimized SpMM kernel. These components jointly enhance bandwidth efficiency and computational intensity during both prefill and decoding phases. Experiments demonstrate that SPDP achieves 1.24–1.37× average speedup (up to 2.51×) over state-of-the-art frameworks such as SpInfer on mainstream GPUs, while preserving model perplexity and enabling up to 25% additional sparsity.

Abstract

The increasing deployment of large language models (LLMs) has magnified the computational and memory bottlenecks of autoregressive decoding, where low compute intensity and bandwidth-bound kernels dominate inference cost. Weight pruning offers a promising remedy, but existing methods remain confined to either static pruning (SP), which permanently removes redundant weights but lacks adaptivity, or dynamic pruning (DP), which adapts to input sparsity but introduces runtime irregularity. This paper presents SPDP, a unified sparse-inference framework that integrates unstructured SP with input-adaptive DP for efficient LLM inference on GPUs. SPDP co-designs a new Tiled-Column-wise Bitmap Compressed (Tiled-CBC) format and two complementary GPU kernels: (1) a CUDA-core spMspV kernel featuring Hybrid Activation-aware Dynamic Shared-Memory Bitmap Decoding (HAD-SMBD) for fine-grained, runtime activation skipping, and (2) a Tensor-Core SpMM kernel optimized for prefill computation. This joint format-kernel design harmonizes static and dynamic sparsity, maintaining bandwidth-efficient memory access and high compute intensity under both phases of LLM inference. Comprehensive evaluations on inference-optimized GPUs demonstrate that SPDP achieves 1.24x-1.37x average speedup (up to 2.51x) over state-of-the- art sparse frameworks such as SpInfer, while matching. perplexity with up to 25% higher sparsity. SPDP advances the inference efficiency-quality Pareto frontier, showing that unified static-dynamic pruning can deliver substantial throughput and performance-per-watt improvements in large-scale LLM serving

Problem

• LLM inference
• weight pruning
• static pruning
• dynamic pruning
• computational bottleneck

Innovation

• unified pruning
• static-dynamic sparsity
• Tiled-CBC format
• GPU kernel co-design
• efficient LLM inference