PolarQuant: Leveraging Polar Transformation for Efficient Key Cache Quantization and Decoding Acceleration

2025-02-01

AI Summary

Large language models (LLMs) incur substantial memory overhead from KV caches, particularly due to outlier values in Key vectors, which hinder low-bit quantization and cause severe accuracy degradation. This work is the first to observe that, under rotary position embedding (RoPE), Key vectors exhibit a natural polar-coordinate structure across dimensions. Leveraging this insight, we propose PolarKV—a polar-coordinate transformation quantization method: Keys are partitioned into 2D subvectors, whose magnitudes (radii) and angles (polar angles) are separately quantized; a lookup-table-based attention mechanism is then designed to approximate query-key dot products. PolarKV effectively transforms the outlier problem into one of smooth angular distribution, drastically reducing KV cache memory (up to 4.2×) and decoding computation. Evaluated on LLaMA-2 and LLaMA-3, it preserves full-precision downstream task performance while accelerating decoding by 1.8–2.3×.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

The KV cache in large language models is a dominant factor in memory usage, limiting their broader applicability. Quantizing the cache to lower bit widths is an effective way to reduce computational costs; however, previous methods struggle with quantizing key vectors due to outliers, resulting in excessive overhead. We propose a novel quantization approach called PolarQuant, which efficiently addresses the outlier challenge. We observe that outliers typically appear in only one of two dimensions, which are rotated together by a specific angle when rotary position embeddings are applied. When represented as two-dimensional vectors, these dimensions exhibit well-structured patterns, with radii and angles smoothly distributed in polar coordinates. This alleviates the challenge of outliers on per-channel quantization, making them well-suited for quantization. Thus, PolarQuant divides key vectors into groups of two-dimensional sub-vectors, encoding them as the corresponding quantized radius and the polar angle, rather than quantizing original key vectors directly. PolarQuant achieves the superior efficiency in KV cache quantization and accelerates the decoding process by turning the query-key inner product into a table lookup, all while maintaining the downstream performance of full-precision models.

Problem

• Large Language Models
• KV Cache Optimization
• Quantization Methods

Innovation

• PolarQuant
• KV Cache Optimization
• Quantization