Diverse activation functions in deep learning

Abstract

We introduce the concept of diverse activation functions, and apply them into Convolutional Auto-Encoder (CAE) to develop diverse activation CAE (DaCAE), which considerably reduces the reconstruction loss. In contrast to vanilla CAE only with activation functions of the same types, DaCAE incorporates diverse activations by considering their cooperation and location. In terms of the reconstruction capability, DaCAE significantly outperforms vanilla CAE and full connected Auto-Encoder, and we conclude rules of thumb on designing diverse activations networks. Based on the high quality of the latent bottleneck features extracted from DaCAE, we demonstrate a satisfying advantage that fuzzy rules classifier performs better than softmax layer in supervised learning. These results could be seen as new research points in the attempts at using diverse activations to train deep neural networks and combining fuzzy inference systems with deep learning.