FP64 Is All You Want, INT8 Is All You Need, FP4/6/8 Is All You Have

2026-09-29

AI Summary

This study addresses the inefficiency of simulating high-precision matrix multiplication on low-precision hardware by formulating the Ozaki scheme as a combinatorial optimization problem. It establishes a theoretical framework and derives a lower bound on the number of GEMM invocations. Methodologically, this work proposes residual modular arithmetic combined with combinatorial optimization algorithms to automatically search for optimal modulus configurations and computation strategies, achieving efficient solutions through GPU parallelization. Furthermore, it introduces the first high-performance FP6 simulation scheme, enabling INT8/FP8/FP4-accelerated high-precision computation on the Blackwell architecture. The proposed approach attains up to 83× the performance of native FP64 execution.

Abstract

Ozaki scheme II emulates FP64 matrix products with INT8 ones through residues modulo pairwise coprime moduli, and variants for FP8 and FP4 have followed. We treat these schemes as one family and pose the choice of a scheme as a combinatorial program that minimizes the number of low-precision GEMMs. Given, for each modulus, a finite set of ways to compute products modulo it from low-precision GEMMs, we find the choice of moduli and ways with the fewest GEMMs, for any format, accumulator and inner dimension, and derive lower bounds on the GEMM count over the whole family. Applied to the formats of current GPUs, the method gives the first FP6 schemes, an FP8 scheme with fewer GEMMs than any previous one, and an FP4 scheme that the bounds show needs the fewest GEMMs of any scheme in the family whose moduli lie in a stated range. Implemented on three Blackwell GPUs, the INT8, FP8 and FP4 schemes run faster than native FP64, up to 83x on B300.

Problem

• low-precision GEMM
• FP64 emulation
• Ozaki scheme
• combinatorial optimization
• matrix multiplication

Innovation

• Ozaki scheme
• combinatorial optimization
• low-precision GEMM
• mixed-precision emulation
• Blackwell GPU