The Sparsity Whisperer

2026-08-06

AI Summary

This work addresses a critical limitation in existing large language model pruning methods, which predominantly rely on large activation magnitudes or output reconstruction while overlooking the role of input-difference-sensitive neurons in MLP layers for discriminating similar samples. To remedy this, the paper introduces output discrepancy as a core pruning signal and proposes three discrepancy-aware methods—Wisp, Wisp+, and Whisper—leveraging input-difference norm scoring, neuron-level discrepancy amplification, and a lightweight regularized differential Hessian reconstruction objective, respectively. These approaches enable efficient zero-update or second-order pruning and seamlessly integrate with existing techniques. Evaluated across Llama 2/3.1 models (7B–405B), they consistently outperform strong baselines such as Wanda and SparseGPT in structured sparsity, downstream tasks, and across multiple model families, effectively advancing the Pareto frontier between accuracy and efficiency.

Abstract

Pruning reduces the inference cost of large language models, but existing criteria primarily preserve large activations or reconstruct layer outputs. We argue that this overlooks a key computation performed by particularly sparsity-sensitive neurons in the MLP up and gate projections: separating similar inputs into dissimilar outputs. This suggests that effective pruning should preserve not only activations, but also the differences between outputs more broadly. We introduce a family of difference-informed pruning methods built upon this principle. Wisp is a first-order, update-free method that scores weights using input-difference norms, and Wisp+ refines this score neuronwise using the input pairs each neuron separates most strongly. Finally, Whisper is a second-order method that uses a lightly regularized difference Hessian as its reconstruction objective. Across Llama 2 and 3.1 models from 7B to 405B parameters, our second-order variant consistently improves over strong reconstruction-based baselines, while our update-free variants improve over activation-aware baselines, especially in constrained settings. The improvements over Wanda and SparseGPT extend to structured sparsity, downstream evaluations, and other model families. Augmenting stronger techniques such as RIA and ALPS with our difference-informed criteria yields further improvements, shifting the overall accuracy-runtime frontier outward at negligible additional cost. These results suggest that preserving output differences is a broadly useful and composable signal for post-training LLM sparsification.

Problem

• pruning
• sparsity
• large language models
• output differences
• MLP

Innovation

• difference-informed pruning
• output separation
• structured sparsity
• post-training sparsification
• Hessian-based pruning