Technology Category
Machine Learning: Hardware-aware MLComputer Vision: Learning & Optimization for CVSearch and Optimization: Learning to Search
Nearly Lossless Adaptive Bit Switching
🤖 AI Summary
To address the challenge of balancing flexibility, efficiency, and accuracy in single-training multi-precision deployment of deep neural networks, this paper proposes a unified multi-precision quantization framework. First, Double Rounding quantization enables near-lossless dynamic switching among INT4/6/8 bit-widths. Second, the Adaptive Learning Rate Scaling (ALRS) mechanism mitigates gradient interference across precision levels. Third, Hessian-Aware Stochastic Bit-switching (HASB) facilitates efficient joint training of mixed-precision models. Our method requires only one quantization-aware training (QAT) pass to support zero-overhead switching across arbitrary integer bit-widths. Evaluated on ImageNet-1K, it significantly outperforms state-of-the-art single-pass multi-precision QAT methods. The framework generalizes effectively to object detection, semantic segmentation, and large language models, reducing storage overhead by up to 75% while incurring negligible accuracy degradation (<0.1% Top-1 drop).
Technology Category
Application Category
📝 Abstract
Model quantization is widely applied for compressing and accelerating deep neural networks (DNNs). However, conventional Quantization-Aware Training (QAT) focuses on training DNNs with uniform bit-width. The bit-width settings vary across different hardware and transmission demands, which induces considerable training and storage costs. Hence, the scheme of one-shot joint training multiple precisions is proposed to address this issue. Previous works either store a larger FP32 model to switch between different precision models for higher accuracy or store a smaller INT8 model but compromise accuracy due to using shared quantization parameters. In this paper, we introduce the Double Rounding quantization method, which fully utilizes the quantized representation range to accomplish nearly lossless bit-switching while reducing storage by using the highest integer precision instead of full precision. Furthermore, we observe a competitive interference among different precisions during one-shot joint training, primarily due to inconsistent gradients of quantization scales during backward propagation. To tackle this problem, we propose an Adaptive Learning Rate Scaling (ALRS) technique that dynamically adapts learning rates for various precisions to optimize the training process. Additionally, we extend our Double Rounding to one-shot mixed precision training and develop a Hessian-Aware Stochastic Bit-switching (HASB) strategy. Experimental results on the ImageNet-1K classification demonstrate that our methods have enough advantages to state-of-the-art one-shot joint QAT in both multi-precision and mixed-precision. We also validate the feasibility of our method on detection and segmentation tasks, as well as on LLMs task. Our codes are available at https://github.com/haiduo/Double-Rounding.
Problem
Research questions and friction points this paper is trying to address.
DNN Quantization
Flexibility and Efficiency
Precision Switching
Innovation
Methods, ideas, or system contributions that make the work stand out.
Dual-Rounding Quantization
Adaptive Learning Rate
Mixed-Precision Training
🔎 Similar Papers
No similar papers found.
