Growing neural networks using orthogonal initialization

Abstract

In the training of neural networks, the architecture is usually determined first and then the parameters are selected by an optimizer. The choice of architecture and parameters is often independent. Whenever the architecture is modified, an expensive retraining of the parameters is required. In this work, we focus on growing the architecture instead of the expensive retraining. There are two main ways to grow new neurons: splitting and adding. In this paper, we propose orthogonal initialization to mitigate the gradient vanish of the new adding neurons. We use QR decomposition to obtain orthogonal initialization. We performed detailed experiments on two datasets (CIFAR-10, CIFAR-100) and the experimental results show the efficiency of our method.