Hardware-Aware Bayesian Neural Architecture Search of Quantized CNNs

Abstract

Advances in neural architecture search (NAS) now provide a crucial assistance to design hardware-efficient neural networks (NNs). This letter presents NAS for resource-efficient, weight-quantized convolutional NNs (CNNs), under computational complexity constraints (model size and number of computations). Bayesian optimization is used to efficiently search for traceable CNN architectures within a continuous embedding space. This embedding is the latent space of a neural architecture autoencoder, regularized with a maximum mean discrepancy penalization and a convex latent predictor of parameters. On CIFAR-100, and without quantization, we obtain 75% test accuracy with less than 2.5M parameters and 600M operations. NAS experiments on STL-10 with 32, 8, and 4 bit weights outperform some high-end architectures while enabling drastic model size reduction (6 Mb–840 kb). It demonstrates our method’s ability to discover lightweight and high-performing models, while showcasing the importance of quantization to improve the tradeoff between accuracy and model size.