Celty: SpMspV GPU Kernel and SIMT Co-Design for Efficient Dual-Sparse LLM Inference

2026-08-02

AI Summary

Existing GPUs struggle to efficiently support scenarios in large language model inference where both weights and activations are simultaneously sparse, primarily due to the lack of dedicated hardware support for Sparse Matrix–Sparse Vector (spMspV) computation. This work proposes a co-optimized solution comprising the RLC-CSC sparse data format, high-performance GPU kernels, and a novel Sparse SIMT Core microarchitecture. For the first time, this approach enables hardware-level sparse vector processing without requiring data reordering, thereby eliminating software overhead from index reconstruction and supporting conflict-free accumulation. Experimental results demonstrate speedups of up to 2.8× over cuBLAS and 2.4× over Flash-LLM; under 70% dual sparsity, the integration with Sparse SIMT Core achieves up to 5.3× acceleration.

Abstract

Large Language Models (LLMs) increasingly rely on sparsity to reduce inference cost, but most prior work targets a single sparsity source-either weight or activation-and optimizes for batched multi-user inference. Dual-sparsity, which combines unstructured weight pruning with runtime activation sparsity, offers a compelling tradeoff among model size, accuracy, and latency for single-user decoding, but formulates as a Sparse Matrix-Sparse Vector (spMspV) workload that existing GPU kernels handle poorly. We propose Celty, a co-designed sparse format, GPU kernel, and SIMT microarchitecture for efficient spMspV in LLM inference. At the kernel level, Celty introduces a Run-Length Compressed CSC (RLC-CSC) format that enables vectorized loading of compressed weight columns and exploits both sparsity sources to skip unnecessary memory accesses, with shared memory used for scattered partial-product accumulation. At the microarchitecture level, the Celty Sparse SIMT Core integrates a pipelined RLC decoder to eliminate software-level index reconstruction and repurposes local register files for conflict-free accumulation-operating directly on the same RLC-CSC format without data layout changes. The Celty GPU kernel achieves up to 2.8x speedup over cuBLAS and 2.4x over Flash-LLM. With the Sparse SIMT Core, speedups reach up to 5.3x over cuBLAS at 70% dual-sparsity.

Problem

• dual-sparsity
• spMspV
• LLM inference
• GPU kernel
• sparse matrix-sparse vector

Innovation

• dual-sparsity
• spMspV
• RLC-CSC
• SIMT co-design
• sparse LLM inference