Accurate Models of NVIDIA Tensor Cores

2025-12-07

AI Summary

NVIDIA Tensor Cores across V100, A100, H100, and B200 GPUs exhibit non-IEEE 754-compliant numerical behaviors—including divergent rounding modes, accumulator bit widths, normalization points, and carry handling—leading to cross-architecture computational irreproducibility. Method: We conduct the first systematic, multi-generation numerical reverse-engineering of Tensor Cores, combining targeted test-vector analysis with floating-point theory to construct a high-fidelity software simulation model. Contribution/Results: The model precisely reproduces mixed-precision matrix multiply-accumulate (MMA) operations at 8-bit, 16-bit, and 19-bit precisions, enabling hardware-level reproducible simulation. It is the first open-source, cross-architecture consistent reference tool for Tensor Core numerical behavior, bridging a critical gap in hardware-aware numerical modeling. This enables rigorous design and verification of mixed-precision algorithms, scientific computing validation, and AI system portability studies.

Technology Category

• Machine Learning: Hardware-aware ML
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Computer Vision: Large Vision Models

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Matrix multiplication is a fundamental operation in for both training of neural networks and inference. To accelerate matrix multiplication, Graphical Processing Units (GPUs) provide it implemented in hardware. Due to the increased throughput over the software-based matrix multiplication, the multipliers are increasingly used outside of AI, to accelerate various applications in scientific computing. However, matrix multipliers targeted at AI are at present not compliant with IEEE 754 floating-point arithmetic behaviour, with different vendors offering different numerical features. This leads to non-reproducible results across different generations of GPU architectures, at the matrix multiply-accumulate instruction level. To study numerical characteristics of matrix multipliers-such as rounding behaviour, accumulator width, normalization points, extra carry bits, and others-test vectors are typically constructed. Yet, these vectors may or may not distinguish between different hardware models, and due to limited hardware availability, their reliability across many different platforms remains largely untested. We present software models for emulating the inner product behavior of low- and mixed-precision matrix multipliers in the V100, A100, H100 and B200 data center GPUs in most supported input formats of interest to mixed-precision algorithm developers: 8-, 16-, and 19-bit floating point.

Problem

• Modeling NVIDIA Tensor Cores for numerical reproducibility
• Studying non-IEEE 754 floating-point behavior in matrix multipliers
• Emulating low-precision inner products across GPU architectures

Innovation

• Software models emulate low-precision matrix multipliers
• Models cover V100, A100, H100, B200 GPU architectures
• Simulate inner product behavior for 8-, 16-, 19-bit formats