MixFP4: Enhancing NVFP4 with Adaptive FP4/INT4 Block Representations

2026-05-29

AI Summary

This work addresses the significant accuracy degradation in FP4 quantization caused by its inability to adapt to heterogeneous numerical distributions across tensor blocks due to a fixed micro-format. To overcome this limitation, the authors propose MixFP4, a block-level adaptive mixed FP4 representation that implicitly encodes the choice between E2M1 and E1M2 formats by repurposing the sign bit of FP8 E4M3 block scaling, thereby incurring zero additional metadata overhead. Both formats are unified into an internal E2M2 representation for computation, eliminating hardware redundancy. The approach remains compatible with standard MMA/GEMM execution paths and integrates a hybrid FP4/INT4 quantization strategy, substantially improving quantization robustness and accuracy on mainstream large language models while adding only 3.1% area and 1.5% power overhead.

Abstract

As large language models continue to scale, fine-grained block-scaled low-precision formats such as NVFP4 are increasingly adopted for their substantial throughput and memory benefits. However, a single FP4 micro-format often mismatches heterogeneous block-level tensor statistics. To address this without changing the standard block-scaled MMA/GEMM execution path, we propose MixFP4, a mixed micro-format extension to NVFP4 that selects between two stored FP4 micro-formats (E2M1 and E1M2) per block. MixFP4 reuses NVFP4's scale hierarchy and encodes the format choice with zero additional metadata by repurposing the sign bit of the FP8 E4M3 block scale. By decoding both micro-formats into a unified internal E2M2 compute representation, MixFP4 avoids datapath duplication. Across representative LLM families, MixFP4 improves FP4 quantization robustness and accuracy over NVFP4 baselines with modest tensor-core overhead (3.1\% area, 1.5\% power).

Problem

• FP4 quantization
• block-scaled formats
• tensor statistics heterogeneity
• low-precision representation
• large language models

Innovation

• MixFP4
• adaptive quantization
• low-precision representation
• block-scaled formats
• tensor-core optimization