Heterogeneity-Aware Microscaling for Efficient Low-Bit LLM Inference

2026-08-04

AI Summary

This work addresses the significant accuracy degradation in low-bit large language model inference caused by the inability of fixed microscaling formats to capture quantization heterogeneity. To overcome this limitation, the authors propose AdaMX—a heterogeneity-aware adaptive microscaling methodology co-designed with a dedicated hardware accelerator. AdaMX is the first to jointly model both cross-block and cross-operand heterogeneity within a microscaling framework and introduces dual block sizes to balance accuracy and energy efficiency. Leveraging adaptive formats, specialized decoders, and compute units, AdaMX enables efficient deployment in 22nm FD-SOI technology. It substantially mitigates the accuracy loss of MXFP4 across 3B–70B models—achieving 83% on commonsense tasks and 82% on MMLU—while reducing memory footprint and energy consumption, and maintains 96% of FP16 accuracy on multimodal tasks.

Abstract

Microscaling (MX) is now the standard for low-bit large language model (LLM) inference. Its 4-bit form MXFP4 still loses substantial accuracy, because existing MX formats fix either the element format or the precision-recovery scheme across blocks, and thus capture only limited quantization heterogeneity. Quantization heterogeneity appears at two levels: 1) across blocks, the preferred element format and precision-recovery scheme vary; 2) across operands, weights and activations require different encoding. We introduce AdaMX (Adaptive Microscaling), a heterogeneity-aware format and accelerator. It selects the precision-recovery scheme per block and the representation per operand, at no increase in equivalent bit width (EBW). One design covers two block sizes, giving a higher-accuracy operating point and a lower-EBW operating point that saves storage. We implement a 22nm FD-SOI AI accelerator prototype with the proposed decoder, computing unit, and quantization logic. Against an otherwise identical MXFP4 accelerator with FP4-only multipliers, AdaMX adds about 1% system energy. At the lower-EBW point, AdaMX stays more accurate than the baseline while lowering both memory footprint and energy. Across LLMs from 3B to 70B, AdaMX removes 83% of the MXFP4 accuracy loss on commonsense and 82% on MMLU, and 43% and 27% of the NVFP4 loss. AdaMX also generalizes to multimodal models. On Gemma-4 12B, it leads MXFP4 on all four vision-language benchmarks and keeps up to 96% of FP16 accuracy.

Problem

• quantization heterogeneity
• low-bit inference
• large language models
• Microscaling
• precision loss

Innovation

• AdaMX
• heterogeneity-aware quantization
• microscaling
• low-bit LLM inference
• adaptive precision recovery