Ozaki Scheme II: A GEMM-oriented emulation of floating-point matrix multiplication using an integer modular technique

2025-04-10

AI Summary

This work addresses the challenge of efficiently simulating high-precision floating-point matrix multiplication (GEMM) under hardware constraints. We propose the first integer modular-arithmetic GEMM simulation framework integrating the Chinese Remainder Theorem (CRT). Unlike Ozaki’s scheme—where precision and computational cost are tightly coupled—our approach decomposes high-precision floating-point multiplication into multiple low-precision integer modular multiply-accumulate operations via CRT, enabling flexible trade-offs between accuracy and efficiency by adjusting the number of moduli. The framework is deeply optimized for INT8 Tensor Cores and incorporates highly tuned GEMM kernels. Experiments show that on RTX 4090 and GH200 GPUs, our FP64 simulation achieves 7.4–9.8 TFLOPS and 56.6–80.2 TFLOPS, respectively—surpassing native FP64 performance. On CPUs, quadruple-precision simulation accelerates up to 2.3× over Ozaki’s method.

Technology Category

• Machine Learning: Hardware-aware ML
• Search and Optimization: Sampling/Simulation-based Search
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Web Mining and Content Analysis: Web data generation and simulation
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

This paper addresses emulation algorithms for matrix multiplication. General Matrix-Matrix Multiplication (GEMM), a fundamental operation in the Basic Linear Algebra Subprograms (BLAS), is typically optimized for specific hardware architectures. The Ozaki scheme is a well-established GEMM-based emulation method for matrix multiplication, wherein input matrices are decomposed into several low-precision components to ensure that the resulting matrix product is computed exactly through numerical operations. This study proposes a novel GEMM-based emulation method for matrix multiplication that leverages the Chinese Remainder Theorem. The proposed method inherits the computational efficiency of highly optimized GEMM routines and further enables control over the number of matrix multiplications, which can enhance computational accuracy. We present numerical experiments featuring INT8 Tensor Core operations on GPUs and FP64 arithmetic on CPUs as case studies. The results demonstrate that FP64 emulation using the proposed method achieves performance levels of up to 7.4 to 9.8 TFLOPS on the NVIDIA RTX 4090 and 56.6 to 80.2 TFLOPS on the NVIDIA GH200, exceeding the measured performance of native FP64 arithmetic. Furthermore, for FP64 computations on CPUs, the proposed method achieved up to a 2.3x speedup in emulating quadruple-precision arithmetic compared to the conventional Ozaki scheme.

Problem

• Emulate floating-point matrix multiplication using integer techniques
• Enhance GEMM efficiency with Chinese Remainder Theorem
• Achieve higher accuracy and speed in FP64 emulation

Innovation

• Emulates FP matrix multiplication via integer modular technique
• Leverages Chinese Remainder Theorem for computational efficiency
• Optimizes GEMM routines for enhanced accuracy and speed