MXAttention: Data-Free Optimal Scaling and Pre-Normalization Quantization for MXFP4 Attention

2026-07-27

AI Summary

This work addresses the challenges of MXFP4 quantization in video generation, where power-of-two scaling induces a truncation–underflow trade-off and softmax row-normalization errors degrade visual quality. The authors propose a data-free post-training quantization framework that, for the first time, establishes a distribution-agnostic universal optimal scaling bound (Qmax = 7.25) and introduces Pre-Normalization Quantization (PNQ) to structurally preserve softmax normalization. By seamlessly integrating Universal Optimal Scaling (UOS) and PNQ into the attention pipeline, the method closes over 95% of the VBench quality gap on Wan2.2 and HunyuanVideo, achieves substantial improvements in frame-level similarity, and ensures that all metric degradations remain below an absolute drop of 0.01.

Abstract

The quadratic cost of attention is a major bottleneck in diffusion-based video generation models. MXFP4 attention provides a promising path toward efficient inference, but direct MXFP4 quantization often degrades generation quality due to two numerical issues: the clipping-underflow trade-off from power-of-two scaling and the row-wise normalization error introduced in the softmax loop. We propose MXAttention, a data-free post-training quantization framework for MXFP4 attention. MXAttention introduces two components: Universal Optimal Scaling (UOS), which exploits the periodic structure of power-of-two microscaling to derive a distribution-independent optimal scaling boundary Qmax=7.25 without calibration or search, and Pre-Normalization Quantization (PNQ), which quantizes unnormalized softmax exponentials before row-wise summation to preserve normalization by construction. Experiments on Wan2.2 and HunyuanVideo show that MXAttention closes at least 95% of the VBench Imaging Quality gap between OCP MXFP4 and FP16, substantially improves frame-level similarity, and preserves FP16-level generation quality with less than 0.01 absolute degradation on all reported VBench metrics. MXAttention also achieves performance competitive with strong NVFP4-based baselines with negligible overhead when fused into the attention pipeline. The implementation is publicly available in MindIE-SD.

Problem

• attention quantization
• MXFP4
• numerical stability
• softmax normalization
• video generation

Innovation

• MXAttention
• MXFP4
• data-free quantization
• optimal scaling
• pre-normalization quantization