WIDE: Boosting Adaptive LLM Inference via Token-level Dynamic Width Pruning

2026-07-30

AI Summary

Existing large language model pruning methods struggle to balance accuracy and inference efficiency under high sparsity: static pruning lacks input adaptability, while dynamic pruning suffers from coarse granularity and limited acceleration. To address this, this work proposes WIDE, the first framework to advance dynamic pruning to the neuron block level, enabling end-to-end differentiable, token-wise dynamic width pruning during both prefill and decoding stages. WIDE adaptively selects groups of attention heads and feed-forward network channels for each token. Through a co-design of pruning and kernels, along with a two-stage training pipeline and hardware-agnostic block-skipping techniques—both inter-block and intra-block—WIDE achieves a 55.1% speedup over the state-of-the-art dynamic depth pruning at 50% sparsity, delivering kernel-level speedups of 1.98× and 4.95× in prefill and decoding phases, respectively, and an overall end-to-end acceleration of 1.68×–1.55×.

Abstract

Pruning is a promising approach for improving the efficiency of LLMs. Existing static structured pruning methods are hardware-friendly and can deliver practical throughput gains, but their input-agnostic computation allocation often causes substantial accuracy degradation under aggressive sparsity. Recent dynamic sparsity methods improve quality retention by adapting computation to individual inputs, yet they remain largely limited to coarse-grained structural decisions and their practical acceleration under real-world inference scenarios remains challenging. To address these challenges, we present WIDE, the first end-to-end differentiable token-level dynamic width pruning framework designed for both prefill and decode scenarios. WIDE enables fine-grained computation allocation by allowing each token to dynamically select attention-head groups and FFN-channel groups, extending dynamic pruning beyond layer-level decisions to neuron-block-level granularity. Through a two-stage training pipeline, WIDE learns effective token-wise sparse execution patterns and achieves substantially better quality retention than existing approaches. To make such fine-grained dynamic pruning practical, we further propose a pruning--kernel co-design framework that decomposes dynamic sparsity acceleration into mask reordering, hardware-agnostic block-level skipping, and hardware-dependent intra-block skipping, enabling efficient execution across different granularities. At 50% sparsity, WIDE provides 55.1% performance boost when compared to the state-of-the-art dynamic depth pruning under calibration-only settings. Under prefill and decoding inference workloads, WIDE achieves close-to-theoretical kernel-level speedups of up to 1.98x for prefill and 4.95x for decoding, as well as 1.68x and 1.55x end-to-end acceleration. Our code is available at https://github.com/EIT-NLP/LLM-Pruning/tree/main/WIDE.

Problem

• LLM pruning
• dynamic sparsity
• token-level computation
• inference efficiency
• accuracy degradation

Innovation

• dynamic width pruning
• token-level sparsity
• neuron-block granularity
• pruning–kernel co-design
• adaptive LLM inference