GPTQ-intrinsic LoRA: A Near-optimal Algorithm for Low-precision Quantization with Low-rank Adaptation

2026-05-31

AI Summary

This work addresses the significant performance degradation of large language models under low-bit quantization, aiming to achieve a better trade-off between compression ratio and accuracy. The authors propose a training-free joint low-bit and low-rank compression method that integrates a low-rank correction term directly into the GPTQ quantization process to minimize layer-wise reconstruction error. They establish, for the first time, an information-theoretic lower bound for this problem and design a near-optimal algorithm matching this bound. By incorporating a Hessian-weighted GPTQ variant, singular vector-guided low-rank structure, and a Bid-Up fixed-grid quantization refinement strategy, the method seamlessly embeds low-rank compensation into quantization without requiring post-processing. Experiments on Qwen3 and DeiT demonstrate substantial improvements over standard GPTQ and GPTQ+LoRA, with the refinement mechanism further enhancing reconstruction fidelity.

Abstract

Post-training quantization is widely used for compressing large neural networks, but aggressive low-bit quantization can significantly degrade model quality. A common remedy is to augment the quantized weights with a low-rank correction, leading to approximations of the form $W\approx Q+LR$. In this paper, we study this low-precision plus low-rank representation through the layer-wise reconstruction objective $\|XW-X(Q+LR)\|_F^2$, where $X$ is a calibration matrix. We establish, to our knowledge, the first information-theoretic lower bounds for this problem under finite-alphabet and bounded low-rank compensation constraints. We then propose GPTQ-intrinsic LoRA, a training-free algorithm that incorporates the low-rank correction directly into a GPTQ-style quantization pass by appropriately augmenting the calibration Hessian. For the choice $L=V_r$, where $V_r$ contains the top right singular vectors of $X$, we prove layer-wise reconstruction error bounds in which the usual GPTQ dependence on $\|X\|_F^2$ is replaced by the rank-$r$ residual $\|X-X_r\|_F^2$, up to regularization terms. Under natural structural assumptions, these bounds match the information-theoretic lower bounds in their dominant scaling, up to constants and mild factors. We also introduce Bid-Up, a fixed-grid quantization refinement step that can be alternated with optimal low-rank compensation with guaranteed non-increasing layer-wise reconstruction error. Experiments on Qwen3 language models and DeiT vision transformers show that GPTQ-intrinsic LoRA improves over GPTQ and GPTQ followed by low-rank compensation, with additional gains from refinement loops.

Problem

• post-training quantization
• low-rank adaptation
• low-precision quantization
• layer-wise reconstruction
• model compression

Innovation

• GPTQ-intrinsic LoRA
• low-precision quantization
• low-rank adaptation
• layer-wise reconstruction
• information-theoretic lower bound