SpenseGPT: Practical One-shot Pruning Enabling Sparse and Dense GEMMs for LLM Inference

2026-06-09

AI Summary

This work addresses the significant accuracy degradation commonly caused by the strict 50% sparsity constraint in semi-structured 2:4 sparsity, as well as the practical limitations of existing relaxed sparsity approaches that rely on specialized compilers or incur runtime overhead, hindering end-to-end acceleration. To overcome these challenges, the authors propose Spense, a hybrid sparse-dense weight format that partitions model weights into 2:4-sparse and dense regions, along with SpenseGPT, a one-shot post-training pruning method. This approach leverages standard sparse and dense GEMM libraries without requiring custom compilation or activation expansion. The method achieves lossless end-to-end decoding acceleration for large language models on B200 GPUs, demonstrating up to 1.2× speedup on Qwen3-32B and Seed-OSS-36B, thereby validating the practical efficacy of semi-structured sparse tensor cores.

Abstract

Semi-structured 2:4 sparsity is widely supported by modern accelerators, providing up to a 2x theoretical speedup. However, its strict 50% sparsity constraint often causes non-negligible accuracy degradation under post-training pruning. Meanwhile, existing relaxed sparsity formats either require specialized compiler support or introduce runtime overheads that limit end-to-end speedup. We propose Spense, a practical hybrid sparse-dense format that splits each weight matrix into a 2:4 sparse region and a dense region. This design relaxes the effective sparsity constraint while remaining compatible with existing high-performance sparse and dense GEMM libraries, avoiding both custom compiler support and input activation expansion. Building on this format, we introduce SpenseGPT, a one-shot post-training pruning method that produces sparse and dense regions. Notably, we show that selecting the right dense regions is important, and we devise two different strategies to choose them. Experiments on Qwen3-32B and Seed-OSS-36B demonstrate that our method achieves up to 1.2x end-to-end decoding speedup on B200 GPUs with FP8 precision, while preserving accuracy. To the best of our knowledge, this is the first one-shot pruning demonstration of real-world end-to-end LLM decoding speedup from semi-structured sparse tensor cores on recent GPUs such as B200s, while maintaining model quality.

Problem

• semi-structured sparsity
• post-training pruning
• LLM inference
• accuracy degradation
• end-to-end speedup

Innovation

• semi-structured sparsity
• one-shot pruning
• hybrid sparse-dense format
• LLM inference acceleration
• post-training pruning