SQ-format: A Unified Sparse-Quantized Hardware-friendly Data Format for LLMs

2025-12-04

AI Summary

Existing low-bit quantization (e.g., W4A8) and sparsification (e.g., 2:4 semi-structured pruning) struggle to jointly achieve high inference accuracy and hardware efficiency for large language models (LLMs), due to either accuracy degradation or poor hardware compatibility. Method: We propose SQ-format—a hardware-friendly, unified sparse-quantized representation that co-designs 2:4 sparsity, W4A8 weight quantization, and sparse activation compression into a single, efficiently executable data format. Leveraging high-precision sparse acceleration, SQ-format compensates for accuracy loss inherent in low-bit computation while maintaining static compression. Contribution/Results: Under static compression, SQ-format achieves Pareto-optimal trade-offs between accuracy preservation and throughput improvement. It sets new state-of-the-art post-training quantization (PTQ) performance across mainstream LLMs, significantly boosting inference throughput on GPUs and emerging AI accelerators. SQ-format establishes a practical, deployable design paradigm for joint sparsity-quantization co-optimization in AI chip architectures.

Technology Category

• Machine Learning: Hardware-aware ML
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Post-training quantization (PTQ) plays a crucial role in the democratization of large language models (LLMs). However, existing low-bit quantization and sparsification techniques are difficult to balance accuracy and efficiency due to the limited hardware support. For example, W4A8 can only achieve the same peak TOPS as W8A8 whereas the GPU-supported sparse data format (2:4 semi-structure sparse) is seldomly adopted due to the loss of accuracy. To bridge this gap, in this paper, we propose the Sparse-Quantized Format (SQ-format), which is a unified data format for quantization and sparsification potentially easily supported by new hardware and existing GPUs. SQ-format makes use of the fact that sparse matrix can be accelerated in high-precision, and low-precision matrix multiplication can also be accelerated accordingly. As such, SQ-format is proposed to achieve Pareto improvement between performance and throughput. This format is particularly suitable for activations with outlier inequality status and makes their static compression possible. We show the state-of-the-art PTQ performance with SQ-format, propose the hardware required to support it, and further offer the design exploration and insights for the next-generation AI accelerators.

Problem

• Balancing accuracy and efficiency in LLM quantization
• Overcoming limited hardware support for low-bit formats
• Enabling static compression for activations with outliers

Innovation

• Unified sparse-quantized format for LLMs
• Hardware-friendly data compression for efficiency
• Pareto improvement in performance and throughput