GLF-Q: Global-Local Feature-based Quantization for Vision Transformers

2026-09-28

AI Summary

This study addresses the severe accuracy degradation in low-bit post-training quantization (PTQ) for Vision Transformers, as well as the overfitting and substantial computational errors inherent in existing methods. To this end, we propose a PTQ framework based on global-local feature alignment. Specifically, this work introduces a pioneering feature-alignment-guided mechanism that eliminates explicit Hessian approximation by leveraging downstream feature supervision instead. Furthermore, a zero-overhead offline Hadamard transform is incorporated to disperse activation outliers, which, combined with the straight-through estimator (STE), facilitates rapid convergence. Experimental results demonstrate that the proposed framework significantly outperforms state-of-the-art methods under 3-bit quantization, exhibits strong out-of-domain robustness, and achieves effective acceleration during 8-bit GPU deployment.

Abstract

Post-training quantization (PTQ) efficiently compresses Vision Transformers (ViTs) without retraining, yet suffers severe accuracy degradation at low bit-widths. Existing optimization-based PTQ methods guide block reconstruction via either soft logits or second-order Hessian proxies. Logit supervision is prone to overfitting on limited calibration data, while Hessian approximations incur structural truncation errors. To address these limitations, we propose \textbf{GLF-Q}, a novel PTQ framework guided by Global-Local Feature alignment. GLF-Q propagates quantized block outputs through downstream full-precision layers to align penultimate-layer representations under local output regularization, providing downstream feature supervision without explicitly approximating the Hessian or using a Taylor expansion. Furthermore, offline Hadamard transformations are introduced with zero runtime overhead to disperse activation outliers across channels, effectively contracting dynamic ranges and reducing quantization errors. Meanwhile, optimizing this loss via a Straight-Through Estimator (STE) achieves rapid convergence, bypassing continuous relaxation rounding formulations such as AdaRound. Extensive experiments across representative ViT architectures demonstrate that GLF-Q with standard uniform quantizers substantially outperforms state-of-the-art methods under 3-bit quantization on image classification. In addition, GLF-Q exhibits strong out-of-domain calibration robustness and achieves speedups under 8-bit GPU deployment.

Problem

• Post-training quantization
• Vision Transformers
• low bit-width
• accuracy degradation
• Hessian approximation

Innovation

• Post-training quantization
• Vision Transformers
• Global-Local Feature alignment
• Hadamard transformation
• Straight-Through Estimator