Understanding LLM Quantization through Activation-Guided Compensation and Orthogonal Residuals

2026-09-18

AI Summary

本文通过激活引导补偿和正交残差方法，解决大语言模型中激进W4A4量化导致的精度损失问题，并提出实用指南优化量化过程。

Abstract

Post-training weight-activation quantization reduces the memory and inference costs of large language models, but aggressive W4A4 quantization remains difficult because activation outliers degrade effective quantization resolution. Although weight optimization, channel-wise scaling, and orthogonal rotation mitigate this problem, the error components they address and their relationship remain unclear. Using an exact decomposition of local weight-activation quantization error into an activation-guided weight compensation term and an orthogonal residual, we bound the residual using persistent channel-wise outlier and regular activation quantities. This decomposition clarifies which error components can be addressed by weight compensation and which require transformation design. We then use the residual bounds to derive practical guidelines for applying randomized Hadamard rotation, sign selection, and channel scaling. In particular, the analysis explains how random signs suppress constructive interference among persistent outlier channels, how sampling multiple sign patterns can improve transformation selection, and how second-moment balancing leads to an $L_2$ scaling rule while a further relaxation recovers SmoothQuant-style $L_\infty$ scaling. We evaluate these guidelines through backpropagation-free configurations across eight Llama and Mistral models, obtaining performance competitive with gradient-trained SpinQuant.

Problem

• LLM Quantization
• Activation Outliers
• Quantization Resolution
• Error Components
• Orthogonal Rotation

Innovation

• Activation-Guided Compensation
• Orthogonal Residuals
• Randomized Hadamard Rotation
• Sign Selection
• Channel Scaling