CONQuER: Hardware-Aware Mixed-Precision Quantisation with Online-Calibrated Surrogates

2026-07-28

AI Summary

This work addresses the performance degradation in deploying deep neural networks caused by the decoupling of quantization and compilation, which leads to suboptimal bit-width configurations for heterogeneous hardware and renders exhaustive hardware-in-the-loop evaluation infeasible at scale. To bridge this gap, we propose the first hardware-aware mixed-precision quantization framework integrated directly into the TOSA compiler stack. Our approach combines NSGA-II multi-objective optimization with a dual-proxy pre-screening mechanism—leveraging cache-bound modeling and feature-space isotropy analysis—and introduces online calibration within IREE to align quantized behavior with actual hardware characteristics. Experiments demonstrate that our method achieves up to 12.19× speedup across mobile CPUs, laptop CPUs, and server GPUs while incurring no more than 1.44% Top-1 accuracy loss, significantly outperforming hardware-agnostic quantization baselines.

Abstract

Deploying deep neural networks on resource-constrained hardware relies on mixed-precision quantisation (MPQ). current deployment toolchains severely fragment this process. Quantisation typically occurs as a hardware-agnostic preprocessing step in front-end frameworks, disconnected from the downstream compilers that generate the physical machine code. This separation leads to suboptimal configurations where assigned bit-widths map poorly to the target machine's heterogeneous hardware execution blocks such as tensor cores and variable-width vector units, incurring severe runtime execution penalties. Furthermore, evaluating these configurations via exhaustive hardware-in-the-loop (HIL) testing is intractable due to the exponentially large search space. We present CONQuER, a unified compiler-integrated infrastructure for hardware-aware MPQ. CONQuER shifts quantisation into the compiler pipeline at the TOSA level, enabling intelligent configuration handling based on compiler support. To evaluate this combinatorial search space of different of model layers within practical compilation budgets, CONQuER couples an NSGA-II evolutionary algorithm with a dual-surrogate prescreening engine. This engine evaluates theoretical cache memory bounds and feature space isotropy to discard non-viable configurations. CONQuER then executes only the strongest candidate policies on hardware via IREE, feeding the execution metrics into an online calibrator. This calibrator aligns the surrogate models with the true hardware behaviour during an NSGA-II evolutionary search. Evaluation across mobile and laptop CPUs, and server GPUs demonstrates that optimal quantisation policies are hardware-dependent. By coupling quantisation with compiler lowering and physical execution, CONQuER discovers Pareto-optimal configurations up to 12.19x faster inference with top-1 accuracy within 1.44% of the unquantised baseline.

Problem

• mixed-precision quantisation
• hardware-aware deployment
• compiler integration
• hardware-in-the-loop testing
• quantisation configuration

Innovation

• mixed-precision quantisation
• hardware-aware compilation
• surrogate modeling
• online calibration
• evolutionary optimization