UniSVQ: 2-bit Unified Scalar-Vector Quantization

2026-06-09

AI Summary

This work addresses the significant accuracy degradation of scalar quantization and the prohibitive computational and memory overhead of vector quantization in 2-bit settings. To bridge this gap, the authors propose UniSVQ—the first unified 2-bit quantization framework that seamlessly integrates scalar and vector quantization. UniSVQ enhances representational capacity by parameterizing codewords as affine transformations over integer lattices, preserving compatibility with integer-only kernels while improving expressiveness. Additionally, it introduces a block-level, data-driven fine-tuning strategy to minimize reconstruction error. Experimental results demonstrate that UniSVQ substantially outperforms existing scalar quantization methods across multiple large language models and zero-shot benchmarks, achieving performance on par with vector quantization while enabling higher inference throughput.

Abstract

Post-training quantization at the 2-bit level enables low-cost deployment and inference acceleration for large language models (LLMs). Scalar quantization (SQ) and vector quantization (VQ) are two primary quantization methods, however, the former suffers from significant performance degradation, and the latter incurs computational and storage overhead. We propose UniSVQ, a unified 2-bit quantization framework that bridges scalar and vector quantization by parameterizing codewords as an affine transform of integer lattices. This structure preserves compatibility with optimized integer kernels while retaining much of VQ's flexibility. We further introduce a data-driven block-wise fine-tuning strategy to directly minimize quantization reconstruction error. Extensive experiments across multiple LLM families and zero-shot benchmarks demonstrate that UniSVQ consistently outperforms state-of-the-art SQ methods and achieves performance comparable to advanced VQ methods, while providing higher inference throughput.

Problem

• post-training quantization
• scalar quantization
• vector quantization
• 2-bit quantization
• large language models

Innovation

• 2-bit quantization
• unified scalar-vector quantization
• affine lattice parameterization
• post-training quantization
• block-wise fine-tuning