Problem
Research questions and friction points this paper is trying to address.
Quantization-aware training
Mixed-precision training
Large language models
Precision allocation
Q-PACE: Dynamic Precision Allocation for Quantization-Aware Training
🤖 AI Summary
This study addresses the performance degradation caused by aggressive quantization during quantization-aware training and the inherent challenges of mixed-precision allocation. To this end, we propose Q-PACE, a method grounded in second-order sensitivity modeling. By leveraging curvature-coefficient-weighted noise mean squared error (MSE) to predict layer-wise loss sensitivity, Q-PACE enables dynamic mixed-precision reallocation with minimal computational overhead, effectively balancing model performance against storage costs. Experimental evaluations on a 4B-parameter large language model demonstrate that Q-PACE significantly outperforms existing approaches, achieving comparable or superior loss performance under tighter memory budgets.
📝 Abstract
Quantization-aware training (QAT) leverages lower-precision arithmetic to reduce the cost of LLM deployment, but aggressive quantization degrades final model performance. A common remedy is mixed-precision training, in which high precision is assigned to some of the layers to maintain performance while keeping the cost constrained. This approach then requires precision assignments for model layers during training. We provide a new approach, called Q-PACE, consisting of a second-order sensitivity model that predicts the loss increase as a sum of quantization noise MSE weighted by per-layer curvature coefficients. During training, we periodically re-compute these coefficients using perturbations across layers, and re-assign precision. Pretraining and supervised fine-tuning experiments on LLMs of up to 4B parameters show that Q-PACE consistently improves over existing mixed-precision training recipes, and achieves comparable loss at substantially lower total memory budgets. We further find that quantization sensitivity is highly predictable by depth and layer type, and its stability during training allows for infrequent, cheap recalibration.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Quantization-Aware Training
Mixed-Precision
Second-Order Sensitivity
Dynamic Precision Allocation
Large Language Models
🔎 Similar Papers
No similar papers found.
