A Comparative Study on Effect of Activation Function Placement in Neural Network Architecture for Regression Problems

Abstract

In recent studies, a modified artificial neural network architecture, where the activation function is placed before the weighted sum, has shown promising results for classification problems like XOR. However, it remains unclear whether the same modified architecture will give better results than conventional architecture for regression problems or not. In this paper, we study the same and compare the modified and the conventional architecture on the basis of mean square error (MSE), R2, and mean absolute error (MAE). From simulations, it is revealed that contrary to the XOR classification problem, there is no significant improvement in the MSE with the modified architecture compared to the conventional architecture. Thus, this study contributes to gain a better understanding of the modified neural network architecture, discussing its limitations and applications. � 2023 IEEE.