Optimizing Physics-Informed Neural Networks with hybrid activation functions: A comparative study on improving residual loss and accuracy using partial differential equations

Abstract

Physics-informed neural networks have bridged the gap between traditional numerical and deep learning based approaches in scientific computing. However, they still face limitations regarding improving convergence, accuracy, and minimizing residual loss, where the activation function plays a crucial role. Traditional activation functions often undergo vanishing gradient problems during backpropagation, highlighting the need for better alternatives for efficient training of Physics Informed Neural Networks. In this paper, new hybrid activation functions were proposed which combine the salient characteristics of traditional activation functions. These activation functions were tested with different network hyperparameters on the Swift–Hohenberg equation, a leading tool for modeling pattern development and evolution in fields like thermal convection, fluid, and temperature dynamics, as well as the Burgers equation. Manual tuning of hyperparameters is employed to critically assess the behavior of new activation functions in different experimental settings. Results show that hybrid activation functions have better learning capabilities compared to traditional activation functions. The GaussSwish hybrid activation function, in particular, proved to be highly effective across different network settings, showing better learning ability in training models for complex problems. This research also reveals that not only activation function but residual points sampled through different Monte Carlo sequences also influence the performance of Physics Informed Neural Networks.