On the convergence of optimizer-activation pairs

The effect of training of deep neural network depends on the selection of the activation function and the optimizer, because the different activation functions lead to distinct loss curvature and the different optimizers will have different performance in distinct curvatures. In this paper, we select different combinations of activation functions and optimizers, seek to select the best combination under the same experiment setting, and take a general discussion for the efficiency of these combinations finally. Moreover, to guarantee fair comparison the hyperparameters tuning is conducted.