The commonly used algorithms to optimize a neural network in supervised learning: Overview, and comparative study

Abstract

The neural network training algorithms or optimizers had already been a lively research topic for several years because they are a crucial part of the neural network structure. That is why; the choice of the training algorithm that can be used to optimize a neural network is one of the most important phases in the neural network's building. Therefore, it is necessary to choose the most adequate optimization algorithm for the desired application, in order to achieve a model that can deal with the best performances. In these papers, we are interested in the training algorithms used in supervised learning. Therefore, we present an overview and a comparative study between the most common algorithms employed to justify the choice of an optimizer, which can deal with high performances.