MixQuant: Adaptive Mixed-Precision Quantization for Large Language Models

2026-07-25

AI Summary

This work addresses the limitations of existing mixed-precision quantization methods, which rely on fixed memory budgets and neglect the influence of upstream layer bit-widths on layer sensitivity. The authors propose a hardware-agnostic, adaptive mixed-precision quantization framework that, for the first time, models layer sensitivity as dependent on upstream quantization configurations. By offline marginalizing over randomly sampled quantization perturbations, the method generates budget-independent sensitivity scores. Coupled with a minimum-bitwidth penalty strategy and greedy search, it enables efficient bit-width allocation requiring only a single calibration pass to support deployment under arbitrary memory constraints. Compatible with mainstream post-training quantizers such as AWQ and GPTQ, the approach significantly outperforms existing baselines on Llama-3.2-3B, Llama-2-7B, and Mistral-7B, achieving up to an 8-point average accuracy gain and reducing perplexity from 12.43 to 10.70 under the tightest memory budget, with minimal deployment overhead rivaling integer linear programming solvers.

Abstract

Mixed-precision quantization improves the accuracy of post-training quantization by allocating higher bitwidths to sensitive layers, but existing methods solve the allocation for a single fixed memory budget. In practice the budget varies across deployments and is unknown at calibration time. Adaptive quantization addresses this with one offline calibration that serves any budget, yet current methods score layer sensitivity in a manner that does not consider its dependency on quantization levels of other layers. We show that a layer's sensitivity depends strongly on the bitwidths of its upstream layers and that this dependence shifts the resulting preferred bit allocation. We propose MixQuant, a technique-agnostic adaptive framework that wraps any base quantizer. MixQuant marginalizes each layer's distortion over random quantized upstream configurations to obtain budget-agnostic scores, calibrates the quantizer's parameters on plans the allocator itself produces, and penalizes allocations that leave layers at the lowest bitwidths. A single greedy pass then serves any budget at deployment. Across Llama-3.2-3B, Llama-2-7B, and Mistral-7B under AWQ and GPTQ, MixQuant outperforms adaptive and mixed-precision baselines in every setting, improving average accuracy by up to 8 points and reducing perplexity from 12.43 to 10.70 at the tightest budget, while matching an ILP solver at negligible deployment cost.

Problem

• mixed-precision quantization
• adaptive quantization
• layer sensitivity
• memory budget
• large language models

Innovation

• adaptive quantization
• mixed-precision
• layer sensitivity
• budget-agnostic
• upstream dependency