Stable FP4 Training via Transposition-Invariant Block Quantization

2026-07-27

AI Summary

This work addresses optimization instability and gradient bias in FP4 training caused by inconsistent scaling due to tensor transposition. To resolve this, the authors propose a transpose-invariant two-dimensional block quantization method that integrates truncation-free scaling with stochastic rounding, ensuring consistency between forward and backward computations. Additionally, they introduce an MXFP8 mixed-precision strategy within the attention mechanism. This approach enables stable large-scale FP4 training for the first time, achieving less than 1.3% degradation in both perplexity and downstream task performance on a 7B dense model and a 30B sparse mixture-of-experts model—results that closely match those of BF16 precision.

Abstract

Reducing training precision is a key lever for improving the e ciency of large language model (LLM) training, but pushing beyond FP8 to 4-bit oating point (FP4) remains challenging due to instability during optimization. We identify a fundamental source of this instability in existing microscaling approaches: scale inconsistency induced by tensor transposition. In conventional 1D block quantization, forward and backward passes assign di erent scaling factors to the same values after transposition, leading to biased and unstable gradient updates. To address this issue, we propose a low-precision training framework based on 2D block FP4 quantization, which enforces transposition-invariant scaling and preserves consistency between forward and backward computations. We further combine this with truncation-free scaling and stochastic rounding to control quantization error and maintain unbiased gradients. To handle the sensitivity of attention mechanisms, we adopt MXFP8 quantization for query and key projections, yielding a practical mixed-precision design. We evaluate our method on dense LLMs up to 7B parameters and a 30B Mixture-of-Experts model, trained on up to 100B tokens. Across all settings, our approach achieves stable end-to-end FP4 training and closely matches BF16 performance, with less than 1.3% degradation in perplexity and downstream accuracy. These results demonstrate that enforcing forwardbackward scaling consistency is su cient to enable practical FP4 training at scale, providing a simple and e ective pathway toward more e cient LLM training.

Problem

• FP4 training
• quantization instability
• tensor transposition
• scaling inconsistency
• large language models

Innovation

• FP4 training
• transposition-invariant quantization
• 2D block quantization
• mixed-precision training
• low-precision optimization