CNN-Based Metrics for Performance Evaluation of Generative Adversarial Networks

In this work, we propose two convolutional neural network (CNN) based metrics, classification score (CS) and distribution score (DS), for performance evaluation of generative adversarial networks (GANs). Though GAN-generated images can be evaluated through manual assessment of visual fidelity, it is prolonged, subjective, challenging, tiresome, and can be misleading. Existing quantitative methods are biased toward memory GAN and fail to detect overfitting. CS and DS allow us to experimentally prove that training of GANs is actually guided by the dataset that it improves with every epoch and gets closer to following the distribution of the dataset. Both methods are based on GAN-generated image classification by CNN. CS is the root mean square (rms) value of three different classification techniques, direct classification (DC), indirect classification (IC), and blind classification (BC). It exhibits the degree to which GAN can learn the features and generate fake images similar to real datasets. DS shows the contrast between the mean distribution of GAN-generated data and the real data. It indicates the extent to which GANs can create synthetic images with similar distribution to real datasets. We evaluated CS and DS metrics for different variants of GANs and compared their performances with existing metrics. Results show that CS and DS can evaluate the different variants of GANs quantitatively and qualitatively while detecting overfitting and mode collapse.