MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference

2026-07-20

AI Summary

This work addresses the significant accuracy degradation in 4-bit quantization of large language models caused by outliers. Existing approaches rely on software-based scaling or dequantization, which incur high computational overhead and struggle to align with hardware-friendly formats. To overcome these limitations, the paper proposes a training-free mixed-precision quantization method that uniquely integrates layer- and column-wise sensitivity analysis with the hardware-native MXINT microscaling format. By dynamically allocating 4-, 6-, or 8-bit mantissa widths at a fine-grained, block-structured level, the method achieves precise control over quantization error without requiring weight rotation or additional training. Under a W4A4KV4 setting, it attains perplexities of 3.77 and 7.63 on WikiText-2 for LLaMA-2-70B and LLaMA-3-8B, respectively, substantially outperforming current state-of-the-art quantization techniques.

Abstract

4-bit quantization enables efficient LLM inference, but suffers from significant accuracy degradation due to outliers. Prior work addresses this problem via data rotation or mixed-precision integer quantization, but often relies on software-managed scaling and frequent dequantization, incurring substantial overhead. Microscaling formats, such as MXINT, eliminate these inefficiencies by encoding scales in hardware, yet remain incompatible with rotation-based methods. Our analysis reveals that outliers vary in severity, from rare extremes to frequent mild deviations, and that quantization sensitivity is unevenly distributed across layers and columns. These insights motivate a fine-grained, sensitivity-guided approach. We introduce MXSens, a training-free method that assigns mixed mantissa bitwidths (4/6/8) based on column- and layer-wise sensitivity, naturally leveraging the block-wise structure of MXINT. MXSens outperforms state-of-the-art quantization methods across a range of models and tasks. Under the W4A4KV4 setting, MXSens achieves perplexities of 3.77 and 7.63 on LLaMA-2-70B and LLaMA-3-8B, respectively, substantially improving over existing baselines on WikiText-2. Our work establishes a new balance between accuracy and resource efficiency for LLM quantization.

Problem

• quantization
• outliers
• mixed-precision
• LLM inference
• accuracy degradation

Innovation

• mixed-precision quantization
• sensitivity-aware
• microscaling formats
• LLM inference
• hardware-efficient