PALS: Percentile-Aware Layerwise Sparsity for LLM Pruning

2026-07-08

AI Summary

This work addresses the limitation of existing one-shot pruning methods for large language models, which typically apply a uniform sparsity ratio across all layers and ignore inter-layer importance variations. The authors propose an adaptive sparsity allocation strategy based on the 99th percentile of activation magnitudes per layer, enabling layer-aware pruning within ±5% of the target sparsity without fine-tuning and with minimal computational overhead. Their findings challenge the common assumption that gradient magnitude is a reliable indicator for weight removal impact. On LLaMA-2-7B at 50% sparsity, the method achieves a WikiText-2 perplexity of 10.96, significantly outperforming uniform pruning (12.92, p<0.001); however, improvements are marginal on LLaMA-3-8B and Mistral-7B.

Abstract

One-shot pruning methods like Wanda and SparseGPT apply the same sparsity ratio to every layer of a transformer, ignoring known variation in layer importance. We propose PALS (Percentile-Aware Layerwise Sparsity), which adjusts per-layer sparsity based on the 99th percentile of activation magnitudes, bounded to $\pm 5\%$ around the target ratio. On LLaMA-2-7B at 50\% sparsity, PALS achieves 10.96 WikiText-2 perplexity versus 12.92 for uniform Wanda (mean over 9 runs, $p < 0.001$). The benefit is architecture-dependent: LLaMA-3-8B shows marginal gains and Mistral-7B shows none. We also find that gradient-based allocation -- the seemingly more principled approach -- produces results worse than random, suggesting that gradient magnitude does not predict the impact of discrete weight removal. PALS adds negligible cost to the pruning pipeline and requires no fine-tuning.

Problem

• LLM pruning
• layerwise sparsity
• layer importance
• one-shot pruning
• sparsity ratio

Innovation

• layerwise sparsity
• activation percentile
• one-shot pruning
• LLM compression
• gradient inefficacy