On the local convergence of ADAM-DPGAN with simultaneous and alternating gradient decent training methods

Abstract

Generative Adversarial Networks (GANs) do not ensure the privacy of the training datasets and may memorize sensitive details. To maintain privacy of data during inference, various privacy-preserving GAN mechanisms have been proposed. Despite the different approaches and their characteristics, advantages, and disadvantages, there is a lack of a systematic review on them. This paper first presents a comprehensive survey on privacy-preserving mechanisms and offers a taxonomy based on their characteristics. The survey reveals that many of these mechanisms modify the GAN learning algorithm to enhance privacy, highlighting the need for theoretical and empirical analysis of the impact of these modifications on GAN convergence. Among the surveyed methods, ADAM-DPGAN is a promising approach that ensures differential privacy in GANs for both the discriminator and the generator networks when using the ADAM optimizer, by introducing appropriate noise based on the global sensitivity of discriminator parameters. Therefore, this paper conducts a theoretical and empirical analysis of the convergence of ADAM-DPGAN. In the presented theoretical analysis, assuming that simultaneous/alternating gradient descent method with ADAM optimizer converges locally to a fixed point and its operator is L-Lipschitz with L < 1, the effect of ADAM-DPGAN-based noise disturbance on local convergence is investigated and an upper bound for the convergence rate is provided. The analysis highlights the significant impact of differential privacy parameters, the number of training iterations, the discriminator’s learning rate, and the ADAM hyper-parameters on the convergence rate. The theoretical analysis is further validated through empirical analysis. Both theoretical and empirical analyses reveal that a stronger privacy guarantee leads to a slower convergence, highlighting the trade-off between privacy and performance. The findings also indicate that there exists an optimal value for the number of training iterations regarding the privacy needs. The optimal settings for each parameter are calculated and outlined in the paper.