High-Performance and Power-Efficient Emulation of Matrix Multiplication using INT8 Matrix Engines

2025-08-05

AI Summary

To address the energy-efficiency and performance bottlenecks of high-precision general matrix multiplication (SGEMM/DGEMM) in deep learning, this work proposes a high-accuracy emulation method leveraging an INT8 low-precision matrix engine. The approach employs tiling-based scheduling, fine-grained quantization, and multi-level accumulation optimization to substantially reduce hardware precision requirements for high-accuracy computation. Evaluated on the NVIDIA GH200 platform, it achieves 1.4× speedup and 43% energy efficiency improvement for DGEMM emulation, and 3.0× speedup with 154% energy efficiency gain for SGEMM emulation—outperforming both native implementations and conventional emulation schemes. The key contribution is the first effective adaptation of dedicated low-precision hardware to double-precision scientific computing, preserving numerical reliability while breaking the energy-efficiency barrier. This establishes a novel paradigm for extending heterogeneous accelerators’ capabilities to high-precision workloads.

Technology Category

• Machine Learning: Hardware-aware ML
• Search and Optimization: Sampling/Simulation-based Search
• Computer Vision: Learning & Optimization for CV

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Recent architectures integrate high-performance and power-efficient matrix engines. These engines demonstrate remarkable performance in low-precision matrix multiplication, which is crucial in deep learning. Several techniques have been proposed to emulate single- and double-precision general matrix-matrix multiplication (SGEMM and DGEMM, respectively) by leveraging such low-precision matrix engines. In this study, we present emulation methods that significantly outperforms conventional approaches. On a GH200 Grace Hopper Superchip, the proposed DGEMM emulation achieves a 1.4x speedup and a 43% improvement in power efficiency compared to native DGEMM for sufficiently large problems. The proposed SGEMM emulation achieves a 3.0x speedup and a 154% improvement in power efficiency compared to native SGEMM for sufficiently large problems. Furthermore, compared to conventional emulation methods, the proposed emulation achieves more than 2x higher performance and superior power efficiency.

Problem

• Emulate high-precision matrix multiplication using INT8 engines
• Improve speed and power efficiency of SGEMM and DGEMM
• Outperform conventional emulation methods and native implementations

Innovation

• Emulates high-precision matrix multiplication using INT8 engines
• Achieves significant speedup and power efficiency improvements
• Outperforms conventional emulation methods in performance