FlexiQ: Adaptive Mixed-Precision Quantization for Latency/Accuracy Trade-Offs in Deep Neural Networks

2025-10-03

AI Summary

To address load fluctuations and the accuracy–latency trade-off in real-time deep neural network inference on hardware accelerators (e.g., NPUs/GPUs), this paper proposes a runtime adaptive mixed-precision quantization method. Our approach dynamically selects low-bit (e.g., 4-bit) computation per channel based on feature value ranges and introduces a lightweight bit-width scheduling mechanism for fine-grained, channel-level precision control. We further design an efficient inference runtime system supporting dynamic bit-width switching. Experiments across 11 vision models demonstrate that: (i) fine-tuned 4-bit models achieve an average accuracy improvement of 6.6%; (ii) when 50% of channels operate at 4-bit while the rest remain at 8-bit, the model incurs only a 0.6% accuracy drop yet delivers a 40% speedup over full 8-bit inference; and (iii) the runtime overhead is negligible. This work bridges practical deployment constraints with accuracy preservation through hardware-aware, adaptive quantization.

Technology Category

• Machine Learning: Hardware-aware ML
• Computer Vision: Large Vision Models
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Neural networks commonly execute on hardware accelerators such as NPUs and GPUs for their size and computation overhead. These accelerators are costly and it is hard to scale their resources to handle real-time workload fluctuations. We present FlexiQ, an adaptive mixed-precision quantization scheme for computer vision models. FlexiQ selectively applies low-bitwidth computation to feature channels with small value ranges and employs an efficient bit-lowering method to minimize quantization errors while maintaining inference accuracy. Furthermore, FlexiQ adjusts its low-bitwidth channel ratio in real time, enabling quantized models to effectively manage fluctuating inference workload. We implemented FlexiQ prototype, including the mixed-precision inference runtime on our custom NPU and GPUs. Evaluated on eleven convolution- and transformer-based vision models, FlexiQ achieves on average 6.6% higher accuracy for 4-bit models with finetuning and outperforms four state-of-the-art quantization techniques. Moreover, our mixed-precision models achieved an efficient accuracy-latency trade-off, with the 50% 4-bit model incurring only 0.6% accuracy loss while achieving 40% of the speedup of the 100% 4-bit model over 8-bit model. Latency evaluations on our NPU and GPUs confirmed that FlexiQ introduces minimal runtime overhead, demonstrating its hardware efficiency and overall performance benefits.

Problem

• Adaptive quantization balances latency and accuracy trade-offs
• Selective low-bitwidth computation minimizes errors in neural networks
• Real-time bitwidth adjustment handles fluctuating inference workloads efficiently

Innovation

• Adaptive mixed-precision quantization for vision models
• Selective low-bitwidth computation for small-range channels
• Real-time bitwidth adjustment for workload fluctuations