Technology Category
Computer Vision: Large Vision ModelsMachine Learning: Calibration & Uncertainty QuantificationNatural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
Mix-QViT: Mixed-Precision Vision Transformer Quantization Driven by Layer Importance and Quantization Sensitivity
🤖 AI Summary
This work addresses the heterogeneity in layer importance and quantization sensitivity across vision transformers (ViTs), including ViT, DeiT, and Swin, under low-bit quantization. To tackle this, we propose an interpretability-driven mixed-precision quantization framework: first, we jointly leverage Layer-wise Relevance Propagation (LRP) and per-layer sensitivity analysis to dynamically allocate bit-widths according to layer-specific precision requirements; second, to mitigate extreme outliers in Post-LayerNorm activations, we introduce a channel-wise clipping-based post-training quantization (PTQ) scheme. The method combines theoretical interpretability with practical deployability. Under PTQ, our approach achieves state-of-the-art accuracy at 3-, 4-, and 6-bit settings; under quantization-aware training (QAT), it attains optimal accuracy-efficiency trade-offs at 2-bit mixed precision. This establishes a novel paradigm for efficient deployment of ViT-family models.
Technology Category
Application Category
📝 Abstract
In this paper, we propose Mix-QViT, an explainability-driven MPQ framework that systematically allocates bit-widths to each layer based on two criteria: layer importance, assessed via Layer-wise Relevance Propagation (LRP), which identifies how much each layer contributes to the final classification, and quantization sensitivity, determined by evaluating the performance impact of quantizing each layer at various precision levels while keeping others layers at a baseline. Additionally, for post-training quantization (PTQ), we introduce a clipped channel-wise quantization method designed to reduce the effects of extreme outliers in post-LayerNorm activations by removing severe inter-channel variations. We validate our approach by applying Mix-QViT to ViT, DeiT, and Swin Transformer models across multiple datasets. Our experimental results for PTQ demonstrate that both fixed-bit and mixed-bit methods outperform existing techniques, particularly at 3-bit, 4-bit, and 6-bit precision. Furthermore, in quantization-aware training, Mix-QViT achieves superior performance with 2-bit mixed-precision.
Problem
Research questions and friction points this paper is trying to address.
Dynamic Precision Adjustment
Visual Transformers
Quantization Sensitivity
Innovation
Methods, ideas, or system contributions that make the work stand out.
Dynamic Precision Adjustment
Layer-wise Quantization
Hierarchical Correlation Propagation
🔎 Similar Papers
2024-07-26arXiv.orgCitations: 0
