ScaleSweep: Accurate NVFP4 Post-Training Quantization of LLMs via Block Scale Initialization

2026-05-30

AI Summary

This work addresses the significant gap between existing AbsMax-based block-wise scale initialization and the optimal solution in NVFP4 quantization, which severely limits the accuracy of 4-bit large language models. To overcome this limitation, we propose ScaleSweep, a method that efficiently searches for the optimal block scale within the first-ever derived theoretical upper and lower bounds on mean squared error (MSE) and weighted MSE (WMSE) for NVFP4. By minimizing reconstruction error, ScaleSweep enables high-accuracy post-training quantization with negligible computational overhead. Evaluations on Llama and Qwen models demonstrate that ScaleSweep substantially outperforms existing approaches, maintaining over 93% of the original model performance even when fully quantizing weights, activations, KV caches, and query states to 4 bits.

Abstract

NVFP4 is a recently introduced hardware-supported FP4 format that improves the fidelity of 4-bit quantization through fine-grained block scales. However, existing NVFP4 scale initialization methods still primarily rely on AbsMax initialization, which leaves a noticeable gap to the optimal solution. To address this, we propose ScaleSweep, a simple and efficient scale optimization method that sweeps over feasible block scale candidates and selects the candidate that minimizes a target objective. We further provide a theoretical analysis of NVFP4 quantization and derive both lower and upper bounds for the required sweep range under mean square error (MSE) and weighted mean square error (WMSE) between the original tensor and the quantized reconstructed tensor. The proposed bounds substantially reduce the sweep space while preserving the optimal candidate, enabling negligible overhead compared with the baseline quantization operators. Experiments on Llama and Qwen models demonstrate that ScaleSweep consistently improves quantization performance over existing initialization methods and further narrows the gap to full precision. In particular, under aggressive end-to-end quantization of weights, activations, KV cache, and query states, ScaleSweep preserves more than 93% of the full-precision performance.

Problem

• NVFP4
• post-training quantization
• block scale initialization
• LLMs
• quantization accuracy

Innovation

• NVFP4
• post-training quantization
• block scale initialization
• ScaleSweep
• LLM quantization