Automated design of neural networks with multi-scale convolutions via multi-path weight sampling

Abstract

The performance of convolutional neural networks (CNNs) relies heavily on the architecture design. Recently, an increasingly prevalent trend in CNN architecture design is the utilization of ingeniously crafted building blocks, e.g., the MixConv module, for improving the model expressivity and efficiency. To leverage the feature learning capability of multi-scale convolution while further reducing its computational complexity, this paper presents a computationally efficient yet powerful module, dubbed EMixConv, by combining parameter-free concatenation-based feature reuse with multi-scale convolution. In addition, we propose a one-shot neural architecture search (NAS) method integrating the EMixConv module to automatically search for the optimal combination of the related architectural parameters. Furthermore, an efficient multi-path weight sampling mechanism is developed to enhance the robustness of weight inheritance in the supernet. We demonstrate the effectiveness of the proposed module and the NAS algorithm on three popular image classification tasks. The developed models, dubbed EMixNets, outperform most state-of-the-art architectures with fewer parameters and computations on the CIFAR datasets. On ImageNet, EMixNet is superior to a majority of compared methods and is also more compact and computationally efficient.