Price of universality in vector quantization is at most 0.11 bit

2026-02-05

AI Summary

This work addresses the practical limitations of vector quantization for compressing large language model weights, which typically relies on input-specific statistics and thus lacks generalizability. The study investigates whether a universal codebook can achieve near-optimal quantization performance without depending on any particular input distribution. By integrating tools from information theory, PCA alignment, water-filling allocation, and high-dimensional spherical covering, the authors theoretically establish—for the first time—that such a universal codebook exists, whose rate-distortion performance is at most 0.11 bits per dimension worse than that of an input-optimized water-filling codebook. This result demonstrates the near-optimality of universal quantization schemes and provides a theoretical foundation for low-precision model storage, although explicit constructions of such codebooks remain an open challenge.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Fast computation of a matrix product $W^\top X$ is a workhorse of modern LLMs. To make their deployment more efficient, a popular approach is that of using a low-precision approximation $\widehat W$ in place of true $W$ ("weight-only quantization''). Information theory demonstrates that an optimal algorithm for reducing precision of $W$ depends on the (second order) statistics of $X$ and requires a careful alignment of vector quantization codebook with PCA directions of $X$ (a process known as"waterfilling allocation''). Dependence of the codebook on statistics of $X$, however, is highly impractical. This paper proves that there exist a universal codebook that is simultaneously near-optimal for all possible statistics of $X$, in the sense of being at least as good as an $X$-adapted waterfilling codebook with rate reduced by 0.11 bit per dimension. Such universal codebook would be an ideal candidate for the low-precision storage format, a topic of active modern research, but alas the existence proof is non-constructive. Equivalently, our result shows existence of a net in $\mathbb{R}^n$ that is a nearly-optimal covering of a sphere simultaneously with respect to all Hilbert norms.

Problem

• vector quantization
• universal codebook
• weight-only quantization
• waterfilling allocation
• low-precision storage

Innovation

• universal codebook
• vector quantization
• waterfilling allocation
• low-precision LLMs
• Hilbert norm covering