Differentiable Architecture Search With Attention Mechanisms for Generative Adversarial Networks

Abstract

Generative adversarial networks (GANs) are machine learning algorithms that can efficiently generate data such as images. Although GANs are very popular, their training usually lacks stability, with the generator and discriminator networks failing to converge during the training process. To address this problem and improve the stability of GANs, in this paper, we automate the design of stable GANs architectures through a novel approach: differentiable architecture search with attention mechanisms for generative adversarial networks ( DAMGAN ). We construct a generator supernet and search for the optimal generator network within it. We propose incorporating two attention mechanisms between each pair of nodes in the supernet. The first attention mechanism, down attention, selects the optimal candidate operation of each edge in the supernet, while the second attention mechanism, up attention, improves the training stability of the supernet and limits the computational cost of the search by selecting the most important feature maps for the following candidate operations. Experimental results show that the architectures searched by our method obtain a state-of-the-art inception score (IS) of 8.99 and a very competitive Fréchet inception distance (FID) of 10.27 on the CIFAR-10 dataset. Competitive results were also obtained on the STL-10 dataset (IS = 10.35, FID = 22.18). Notably, our search time was only 0.09 GPU days.