Low-Dimensional High-Leverage Subspace Optimization: Beyond Full-Parameter Coupled Training for Neural Network Quantization

2026-08-04

AI Summary

This work addresses the severe accuracy degradation in low-bit quantization of compact networks, which stems from neglecting the heterogeneity of parameter subspaces. The study identifies that the affine parameters of normalization layers span a low-dimensional, high-leverage subspace capable of fully compensating for the channel-wise affine component of quantization distortion. Building on this insight, the authors propose Normalization Affine Preconditioning (NAP): in post-training quantization (PTQ), it fine-tunes only this critical subspace to enhance quantization compatibility; in quantization-aware training (QAT), it decouples feature learning and numerical calibration via alternating optimization, augmented with distillation guidance and theoretical error-bound analysis. NAP substantially recovers collapsed low-bit performance on ImageNet and CIFAR-100, surpassing full-parameter QAT at minimal tuning cost and consistently improving reconstruction-based PTQ.

Abstract

Low-bit quantization suffers severe accuracy degradation on compact networks, rooted in the dominant full-parameter coupled training paradigm that ignores parameter subspace heterogeneity. Their limited feature redundancy leaves little room to absorb quantization errors. Conventional pipelines adopt monolithic optimization: PTQ reconstructs fixed pretrained models without improving inherent quantization friendliness; QAT updates all parameters jointly, suffering from gradient coupling between backbone weights and calibration parameters. In this paper, we identify normalization affine parameters as a low-dimensional high-leverage subspace dominating quantization robustness, and propose Normalization Affine Preconditioning (NAP) for targeted subspace optimization. For PTQ, NAP freezes backbone weights and fine-tunes only affine parameters under the target fake-quantization graph on full-precision models, proactively boosting quantization friendliness before downstream reconstruction. For QAT, we introduce an alternating QAT-NAP schema that decouples feature learning and numerical calibration, breaking the performance ceiling of saturated joint training. Theoretical analysis confirms BN affine parameters fully cancel the channel-wise affine component of quantization distortion, while nonlinear rounding and clipping residuals form the irreducible error boundary; distillation-guided NAP acts as directional flatness optimization, projecting teacher-student logit mismatch onto the restricted subspace. Experiments on ImageNet and CIFAR-100 show NAP recovers severely collapsed low-bit quantization, consistently boosts reconstruction-based PTQ, and outperforms saturated full-parameter QAT with negligible tuning cost. This work reveals the principle of targeted low-dimensional subspace optimization, offering a new perspective beyond full-parameter coupled training for efficient deep learning.

Problem

• neural network quantization
• low-bit quantization
• parameter subspace heterogeneity
• quantization accuracy degradation
• compact networks

Innovation

• low-dimensional subspace optimization
• neural network quantization
• normalization affine parameters
• decoupled training
• quantization-aware training