Commercial Evaluation of Zero-Skipping MAC Design for Bit Sparsity Exploitation in DL Inference

2024-02-29

IEEE/IFIP International Conference on Very Large Scale Integration of System-on-Chip

AI Summary

To address hardware energy-efficiency bottlenecks in deep learning inference, this work proposes and commercially validates the Zero-skipping Multiply-Accumulate unit (OzMAC), which dynamically exploits bit-level sparsity in both weights and activations. Implemented in TSMC N5 technology, OzMAC features a custom, frequency-scalable MAC architecture supporting INT4/8/16 multi-precision computation—the first post-synthesis commercial feasibility evaluation across multiple precisions and operating frequencies. Experimental results show that, at INT8 precision, OzMAC reduces area by 21%, power consumption by 70%, and energy per operation by 28% versus a conventional baseline. While maintaining throughput, it achieves a 30% improvement in end-to-end energy efficiency—surpassing the theoretical efficiency ceiling of binary MACs. This work demonstrates that high-bitwidth sparsity is not only prevalent but also highly exploitable in hardware, establishing a new paradigm for energy-efficient AI accelerators.

Technology Category

• Machine Learning: Hardware-aware ML
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Search and Optimization: Learning to Search

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Energy management for devices in mobile Web and WoT environments
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

General Matrix Multiply (GEMM) units, consisting of multiply-accumulate (MAC) arrays, perform bulk of the computation in deep learning (DL). Recent work has proposed a novel MAC design, Bit-Pragmatic (PRA), capable of dynamically exploiting bit sparsity. This work presents OzMAC (Omit-zero-MAC), a modified re-implementation of PRA, but extends beyond earlier works by performing rigorous post-synthesis evaluation against binary MAC design across multiple bitwidths and clock frequencies using TSMC N5 process node to assess commercial implementation potential. We demonstrate the existence of high bit sparsity in eight pretrained INT8 DL workloads and show that 8-bit OzMAC improves all three metrics of area, power, and energy significantly by 21%, 70%, and 28%, respectively. Similar improvements are achieved when scaling data precisions (4, 8, 16 bits) and clock frequencies (0.5 GHz, 1 GHz, 1.5 GHz). For the 8-bit OzMAC, scaling its frequency to normalize the throughput, it still achieves 30% improvement on both power and energy.

Problem

• Zero-skipping MAC
• Deep Learning Efficiency
• Energy Consumption Reduction

Innovation

• OzMAC
• Zero-skipping MAC
• Energy efficiency