A Systolic Array Architecture for Nonlinear Activation Functions and Softmax Computation using Chebyshev Polynomials

2026-08-05

AI Summary

This work addresses the inefficiency in neural network accelerators caused by dedicated, non-shareable hardware units for activation functions and softmax. To overcome this limitation, the authors propose a unified activation unit architecture based on systolic arrays, which—by integrating Chebyshev polynomial approximation for the first time—enables efficient sharing between univariate activation functions (e.g., tanh) and multivariate softmax computation. Experimental results demonstrate that the proposed design significantly reduces hardware overhead while maintaining high accuracy: the mean absolute error for tanh approximation is reduced by 71%, with area and power savings of 4.6% and 5.1%, respectively; for softmax, the KL divergence is lowered by 44.6% and 79.0% compared to CORDIC- and piecewise-linear-based approaches.

Abstract

Neural Network Accelerators have gained popularity in recent years due to their greater efficiency than CPU-based platforms. Often, these accelerators utilize different hardware units for univariate activation functions, such as tanh, and the multivariate softmax, thereby missing opportunities for resource sharing between them. In this paper, we describe a novel systolic array-based activation unit architecture that supports multiple univariate activation functions as well as the softmax function. By utilizing Chebyshev polynomial approximations, our activation function unit achieves up to 71% lower mean absolute error for tanh compared to a CORDIC baseline, while using 4.6% less area and 5.1% less power. Our softmax approximation enables a 44.6% and 79.0% lower KL divergence compared to CORDIC and a piecewise-linear approximation, respectively.

Problem

• Neural Network Accelerators
• Activation Functions
• Softmax
• Resource Sharing
• Hardware Efficiency

Innovation

• systolic array
• Chebyshev polynomials
• activation function
• softmax approximation
• neural network accelerator