LSBO-NAS: Latent Space Bayesian Optimization for Neural Architecture Search

From the perspective of data stream, neural architecture search (NAS) can be formulated as a graph optimization problem. However, many state-of-the-art black-box optimization algorithms, such as Bayesian optimization and simulated annealing, operate in continuous space primarily, which does not match the NAS optimization due to the discreteness of graph structures. To tackle this problem, the latent space Bayesian optimization NAS (LSBO-NAS) algorithm is developed in this paper. In LSBO-NAS, the neural architectures are represented as sequences, and a variational auto-encoder (VAE) is trained to convert the discrete search space of NAS into a continuous latent space by learning the continuous representation of neural architectures. Hereafter, a Bayesian optimization (BO) algorithm, i.e., the tree-structure parzen estimator (TPE) algorithm, is developed to obtain admirable neural architectures. The optimization loop of LSBO-NAS consists of two stages. In the first stage, the BO algorithm generates a preferable architecture representation according to its search strategy. In the second stage, the decoder of VAE decodes the representation into a discrete neural architecture, whose performance evaluation is regarded as the feedback signal for the BO algorithm. The effectiveness of the developed LSBO-NAS is demonstrated on the NAS-Bench-301 benchmark, where the LSBO-NAS achieves a better performance than several NAS baselines.