From 2:4 to 8:16 sparsity patterns in LLMs for Outliers and Weights with Variance Correction

2025-07-03

AI Summary

Existing structured sparsity methods for large language models (LLMs) suffer from limited flexibility, sensitivity to outlier weights, and substantial accuracy degradation. Method: We propose a novel semi-structured sparsity paradigm—8:16 sparsity—retaining 8 out of every 16 weights, offering higher sparsity freedom and robustness than conventional 2:4 patterns under identical memory overhead. Our approach integrates saliency-guided structured pruning, variance correction, and SmoothQuant-inspired weight equalization to mitigate distribution shift induced by sparsification. Contribution/Results: Experiments demonstrate that, under identical memory constraints, our method surpasses both the uncompressed baseline and state-of-the-art sparse counterparts across multiple benchmarks. Notably, it achieves the first reported accuracy gain—i.e., performance inversion—at 50% sparsity density, establishing a new paradigm for efficient LLM deployment.

Technology Category

Machine Learning: Large Multimodal Models (LMMs)Natural Language Processing: (Large) Language ModelsComputer Vision: Large Vision Models

Application Category

Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMsGraph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphsSearch and Retrieval-Augmented AI: Large language models for search

Abstract

As large language models (LLMs) grow in size, efficient compression techniques like quantization and sparsification are critical. While quantization maintains performance with reduced precision, structured sparsity methods, such as N:M sparsification, often fall short due to limited flexibility, and sensitivity to outlier weights. We explore 8:16 semi-structured sparsity, demonstrating its ability to surpass the Performance Threshold-where a compressed model matches the accuracy of its uncompressed or smaller counterpart under equivalent memory constraints. Compared to 2:4 sparsity, 8:16 offers greater flexibility with minimal storage overhead (0.875 vs. 0.75 bits/element). We also apply sparse structured patterns for salient weights, showing that structured sparsity for outliers is competitive with unstructured approaches leading to equivalent or better results. Finally, we demonstrate that simple techniques such as variance correction and SmoothQuant like weight equalization improve sparse models performance.

Problem

• Enhancing LLM compression via flexible 8:16 sparsity patterns
• Addressing outlier sensitivity in structured sparsity methods
• Improving sparse model performance with variance correction techniques

Innovation

• 8:16 semi-structured sparsity for flexibility
• Structured sparsity patterns for salient weights
• Variance correction enhances sparse models