On the Role of Discriminator in GAN: A Convex Duality Approach


Farzan Farnia
Stanford University


Mohammad Alizadeh
Abstract: Generative adversarial network (GAN) is a minimax game between a generator mimicking the true model and a discriminator distinguishing the samples produced by the generator from the real training samples. Given a discriminator trained over the entire space of functions, this game reduces to finding the generative model minimizing a divergence score, e.g. the Jensen-Shannon (JS) divergence, to the data distribution. However, in practice the discriminator is trained over smaller function classes such as convolutional neural nets. Then, a natural question is how the divergence minimization interpretation changes as we constrain the discriminator. In this talk, we address this question by developing a convex duality framework for analyzing GANs. We show GAN in general can be interpreted as minimizing divergence between two sets of probability distributions: generative models and discriminator moment matching distributions. We prove that this interpretation applies to a wide class of existing GAN formulations including vanilla GAN, f-GAN, Wasserstein GAN, Energy-based GAN, and MMD-GAN. As a byproduct, we use the convex duality framework to explain why regularizing the Lipschitz constant of discriminator can dramatically improve models learned by vanilla and energy-based GANs and help them achieve state-of-the-art performance over multiple benchmark tasks. We empirically explore the power of this regularization scheme for improving training performance in various GAN formulations.

Biography: Farzan Farnia is a final-year PhD student in the department of electrical engineering at Stanford University where he is advised by David Tse. Farzan received his masters degree in electrical engineering from Stanford University in 2015 and prior to that two bachelors degree in electrical engineering and mathematics from Sharif University of Technology in 2013. His research interests include information theory, statistical learning theory, and convex optimization. He has been the recipient of a Stanford graduate fellowship (Sequoia Capital fellow) from 2013-2016 and the Numerical Technology Founders Prize as the second top performer of Stanford electrical engineering PhD qualifying exams in 2014.