SCAN: Streamlined Composite Activation Function Unit for Deep Neural Accelerators

Abstract

Transcendental nonlinear function design in deep neural accelerators principally concerns performance parameters such as area, power, delay, and throughput. Neural hardware demands resource-intensive blocks such as adders, multipliers, and nonlinear activation functions. This work addresses the issues related to the implementation of the activation function for deep neural accelerators. The proposed design implements an activation function unit with the help of stochastic computing along with clock gating techniques to reduce the active power dissipation in the hardware. But a complete deep neural network uses various activation functions in the hidden layers. To overcome the problem of implementing individual hardware design corresponding to each activation function, we have designed the streamlined composite activation function unit for neural accelerators (SCAN), which implements hyperbolic tangent and ReLU activation functions. The proposed method using stochastic computing along with clock gating is compared with other states of the art. The area is reduced by approximately 74.14% as compared to that of CORDIC-based design. While implementing a single neuron, both area and power are reduced by manifold, enhancing the performance of deep neural accelerators. Testing accuracy and inference time are calculated using the benchmark dataset (MNIST) on AlexNet architecture. Testing accuracy in the proposed implementation is increased by 1.08% , and loss is reduced by 40.66%. © 2022, The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature.