GAN Training With Kernel Discriminators: What Parameters Control Convergence Rates?

Abstract

Generative Adversarial Networks (GANs) are widely used for modeling complex data. However, the dynamics of the gradient descent-ascent (GDA) algorithms, often used for training GANs, have been notoriously difficult to analyze. We study these dynamics in the case where the discriminator is kernel-based and the true distribution consists of discrete points in Euclidean space. Prior works have analyzed the GAN dynamics in such scenarios via simple linearization close to the equilibrium. In this work, we show that linearized analysis can be grossly inaccurate, even at moderate distances from the equilibrium. We then propose an alternative non-linear yet tractable second moment model. The proposed model predicts the convergence behavior well and reveals new insights about the role of the kernel width on convergence rate, not apparent in the linearized analysis. These insights suggest certain shapes of the kernel offer both fast local convergence and improved global convergence. We corroborate our theoretical results through simulations.