The Most Used Activation Functions: Classic Versus Current

Abstract

This paper is an overview of the most used activation functions, classic functions and current functions as well. When we say classic, we mean the first activation functions, the most popular and used in the past. But due to their disadvantages appeared other new activation functions that we refer them as current. These most influential functions are among the most known artificial intelligence activation functions in the research of Machine learning and Deep Learning as well. With each function, we provide a brief description of the activation function, discuss its impact and show the domain where it is applicable, its advantages and disadvantages and more details to have an overview. These functions cover more issues like vanishing gradient, exploding gradient when we are using Gradient Descent and so on. These solutions to these issues are all among the most important topics in Artificial Intelligence research and development part.