GAN: Generative Adversarial Networks

A generator learns to produce data by training against a classifier that tries to catch its fakes.

Goodfellow and colleagues’ 2014 method fits a generative model without ever writing down its probability density. The generator’s only training signal is the gradient of a second network’s verdict on its samples.

The page derives the paper’s two main results with worked numbers and trains a small GAN in the browser, where you can switch between the paper’s two generator losses and watch one of them stall.

Why the density was the hard part

A generative model of images is a program that outputs new images resembling a training set. The standard way to fit one in 2014 was maximum likelihood: define a probability density $p(\mathbf{x})$ over images, then adjust its parameters so the training images get high probability. That requires computing $p(\mathbf{x})$, or at least its gradient, for every training image.

For the deep models of the time this was intractable. A restricted Boltzmann machine on 28×28 binary images defines $p(\mathbf{x})$ up to a normalizing constant, the partition function, which sums over every joint setting of its 784 visible and $H$ hidden binary units: $2^{784+H}$ terms. Its gradient can only be estimated with Markov chain Monte Carlo, and the paper’s related-work section notes that mixing (how fast the chain moves between modes) “poses a significant problem” for those methods. Directed models had the matching problem of inference over latent variables. The variational autoencoder, published a few months earlier, handled that with a learned inference network and a bound on the likelihood.

Generative Adversarial Nets (Goodfellow, Pouget-Abadie, Mirza, Xu, Warde-Farley, Ozair, Courville and Bengio, Université de Montréal) drops the density. The model is a network that turns random noise into a sample, and nothing ever evaluates how probable a sample is. The training signal comes from a second network, a classifier trained to tell real examples from generated ones. The paper proves that if both networks were unlimited in capacity and trained to their optima, the generator’s samples would follow the data distribution exactly. Training needs only backpropagation, and sampling needs one forward pass.

The sections below go in dependency order: the game and its value function; the generator as a warp of noise; the optimal discriminator; what the generator minimizes against it; how Algorithm 1 approximates the game; why the generator’s loss was changed; a live 1-D GAN; the failure modes; and what the experiments do and do not show.

Two networks, one value function

The paper’s own analogy is counterfeiters and police. The counterfeiters print fake currency and try to spend it without being caught; the police try to tell fake bills from real ones. Each side improves against the other, and the contest ends when the fakes are indistinguishable from real money. The analogy breaks in one place: the counterfeiters here never see a real bill. They learn only from how the police react to their fakes.

The generator $G$ is a neural network that takes a noise vector $\mathbf{z}$, drawn from a fixed prior $p_{\mathbf{z}}$, and outputs a sample $G(\mathbf{z})$ in data space. The discriminator $D$ takes a data-space input $\mathbf{x}$ and outputs one number in (0, 1), its estimate of the probability that $\mathbf{x}$ came from the training data rather than from $G$. Both are multilayer perceptrons. They share one objective, the value function $V$, which $D$ maximizes and $G$ minimizes:

The first term is large when $D$ gives real examples a score near 1. The second is large when $D$ gives generated samples a score near 0, since then $1 - D$ is near 1. Both logs are at most 0, so the best $D$ can do is push $V$ up toward 0. The generator appears only in the second term, and it lowers $V$ by making $D(G(\mathbf{z}))$ larger.

With numbers: say on one real example $D$ outputs 0.9 and on one fake it outputs 0.2. That pair contributes

A sharper discriminator that scores them 0.99 and 0.01 gets −0.010 − 0.010 = −0.020, closer to the maximum of 0. A better generator whose fake now scores 0.9 drives the second term to log 0.1 = −2.303. All logs on this page are natural logs.

The paper calls this a “two-player minimax game”. Later writing, including Goodfellow’s own 2016 tutorial, describes the solution as a Nash equilibrium and a saddle point of $V$: a minimum along the generator’s parameters and a maximum along the discriminator’s. Those terms are used below where they help, but they are not in the 2014 paper, which frames the solution as the global minimum of a criterion $C(G)$ derived in a later section.

The generator warps noise

The generator is deterministic: the same $\mathbf{z}$ always gives the same output. All the randomness in its samples comes from $\mathbf{z}$. The paper puts it this way: “The generator G implicitly defines a probability distribution $p_g$ as the distribution of the samples $G(\mathbf{z})$ obtained when $\mathbf{z} \sim p_{\mathbf{z}}$.” That distribution is the pushforward of the prior through $G$. You can draw from it by sampling $\mathbf{z}$ and running the network, but no formula for $p_g(\mathbf{x})$ is ever written down, which is what “implicit” means.

A smooth map shapes a distribution by stretching some regions of noise space and compressing others. In one dimension the standard change-of-variables rule makes this exact. It is background, not something the paper states, and a GAN never computes it:

Take $z$ uniform on (0, 1) and $G(z) = z^2$. Then

which is 5.0 at $x = 0.01$ and 0.56 at $x = 0.81$. Near $z = 0$ the slope $G'(z) = 2z$ is small, so a wide interval of $z$ lands in a narrow interval of $x$ and probability piles up. The paper’s Figure 1 caption says the same thing: “G contracts in regions of high density and expands in regions of low density of $p_g$.” Normalizing flows are generative models that use this rule directly, which forces them to be invertible with a computable Jacobian; a GAN’s generator needs neither property.

Figure 1 shows the warp. Evenly spaced noise values on the bottom line are carried by threads to their outputs on the top line. With the slider at 0, $G$ is a stretched identity and the output is as flat as the input. Drag it to the right and watch where the threads converge: under the two modes of the target, where the curve of $G$ is flat.

In the paper’s MNIST experiment the same idea runs in 100 dimensions. The released configuration draws $\mathbf{z}$ uniformly from $[-\sqrt3, \sqrt3]^{100}$ and passes it through two 1200-unit ReLU layers to 784 sigmoid outputs, one per pixel. So for a batch of 100, $\mathbf{z}$ is a 100×100 matrix and $G(\mathbf{z})$ is 100×784. Noise enters only at the bottom layer (§5). The uniform prior is the one in the paper’s Figure 1 caption and in that config; the paper’s text does not fix one prior for every experiment.

The optimal discriminator

Freeze the generator, so $p_g$ is some fixed distribution, and ask which discriminator maximizes $V$. The paper’s Proposition 1 answers exactly. First, rewrite $V$ as one integral over data space. Real examples arrive with density $p_{\text{data}}(x)$ and fakes with density $p_g(x)$, and because $p_g$ is defined as the distribution of $G(\mathbf{z})$, the expectation over $\mathbf{z}$ becomes an expectation over $x$ with no Jacobian factor:

The integrand at a point $x$ depends only on the single number $D(x)$, so the whole integral is maximized by maximizing it separately at every $x$. Write $a = p_{\text{data}}(x)$ and $b = p_g(x)$. The function $f(y) = a\log y + b\log(1-y)$ has derivative

and its second derivative $-a/y^2 - b/(1-y)^2$ is negative, so this is a strict maximum. It lies inside (0, 1) whenever $a$ and $b$ are both positive, since $f$ goes to $-\infty$ at either end. Substituting back:

At each point, the best discriminator reports the share of probability there that belongs to the data. Pick a point where $p_{\text{data}}(x) = 0.4$ and $p_g(x) = 0.1$. Then $D^*(x)$ = 0.4/0.5 = 0.8. If the generator improves until $p_g(x) = 0.4$ as well, $D^*(x) = 0.5$, because a sample landing at that point is equally likely to be real or fake. Where only data lives, $D^* = 1$; where only fakes live, $D^* = 0$. The paper adds that $D$ need not be defined outside the union of the two supports, where the integrand is zero anyway.

Rearranged, (2) is the density ratio between real and fake:

At the example point, 0.8/0.2 = 4 = 0.4/0.1. The discriminator never learns either density, but its output encodes their ratio. The paper also reads D’s objective as logistic regression for $P(\text{real} \mid x)$. That reading holds because (1) weights real and fake examples equally, which amounts to a prior of one half on each class. A classifier trained on one real example for every three fakes would instead converge to

at the same point. The ratio form above holds either way.

Figure 2 plots (2) for a two-mode data distribution and a generator distribution of the same shape, shifted to the right. The probe reads out $p_{\text{data}}$, $p_g$ and $D^*$ at one point; drag it across a mode and watch $D^*$ track the local mix. Then drag the training slider to 100% and the violet curve flattens onto the dashed line at one half.

A related idea already existed. Noise-contrastive estimation (NCE, Gutmann and Hyvärinen 2010) fits a density model by training it to classify data against samples from a fixed noise distribution. The paper points out that because NCE’s noise is fixed, learning slows sharply once the model is roughly right. In a GAN the samples being classified against the data come from the generator, and they move toward the data during training, so the classification stays informative.

What the generator minimizes

Substitute the optimal discriminator (2) into (1). What remains depends only on $G$; the paper calls it the virtual training criterion $C(G)$:

To see what this measures, introduce the average distribution $M = (p_{\text{data}} + p_g)/2$. The denominators in (4) equal $2M$, so each log splits into a log-ratio against $M$ minus $\log 2$. The expectation of $\log(P/M)$ under $P$ is the Kullback-Leibler divergence $\mathrm{KL}(P\,\|\,M)$, which is zero when $P = M$ and positive otherwise. The two $-\log 2$ terms add to $-\log 4$:

The sum of those two KL terms is twice the Jensen-Shannon divergence (JSD), a symmetric measure of how different two distributions are, defined as the average KL of each to their mixture:

The JSD is non-negative and zero only when the two distributions are equal. So Theorem 1 follows: the global minimum of $C(G)$ is $-\log 4 \approx -1.386$, reached if and only if $p_g = p_{\text{data}}$. At that point (2) gives $D^* = \tfrac12$ everywhere, and plugging one half into (4) gives log ½ + log ½ = −log 4 directly.

The other end of the range follows from a bound the paper does not state, $\mathrm{JSD} \le \log 2$ (Lin 1991). When the two distributions do not overlap at all, $M$ equals half of $p_{\text{data}}$ wherever the data lives, so $\log(p_{\text{data}}/M) = \log 2$ there and the first KL term is exactly $\log 2$. The second is the same, so

and $C$ ranges from −1.386 (identical) to 0 (disjoint).

Figure 3 computes (5) for a Gaussian $p_{\text{data}}$ with standard deviation 0.5 and a Gaussian $p_g$ you can shift and widen. At a shift of +1.5 (three standard deviations) each KL term is 0.527 and $C = -0.333$. Move the shift to 0 with the width at ×2 and $C$ stops at −1.201: matching the mean is not enough. Then push the shift past +3 and read the slope: at +4 the KL terms are both 0.693 (that is, $\log 2$), $C$ is 0.000 to three decimals, and the slope along the shift is 0.001, against 0.796 at a shift of +0.5.

A generator whose samples sit where the data has no mass gets almost no signal from this objective about which way to move, because $C$ changes very little as $p_g$ slides. That observation is the starting point of Wasserstein GAN, discussed at the end.

Everything so far lives in the space of distributions. Proposition 2 states the convergence result: if $G$ and $D$ have enough capacity, if $D$ reaches its optimum at every step, and if $p_g$ itself is updated to improve the criterion, then $p_g$ converges to $p_{\text{data}}$. The proof uses the fact that $C$ is convex in $p_g$ with the unique minimum from Theorem 1. The paper then says what this does not cover: “In practice, adversarial nets represent a limited family of $p_g$ distributions via the function $G(\mathbf{z};\theta_g)$, and we optimize $\theta_g$ rather than $p_g$ itself. Using a multilayer perceptron to define G introduces multiple critical points in parameter space.” The result guarantees that the distribution converges in function space; it says nothing about gradient descent on the weights.

Algorithm 1: alternate the updates

The theory assumes $D$ is at its optimum whenever $G$ moves. Solving the inner $\max_D$ to completion before every generator step would work in principle, and the paper gives two reasons not to: it is “computationally prohibitive, and on finite datasets would result in overfitting.” Algorithm 1 instead alternates $k$ steps of minibatch gradient ascent on D with one step of descent on G. D then stays near its optimum “so long as G changes slowly enough.” The experiments used $k = 1$, “the least expensive option”, and SGD with momentum.

Alternating updates on a min-max objective do not behave like descent on a single loss. Figure 4 uses a two-parameter toy game from the later GAN literature: the generator’s parameter $x$ descends and the critic’s parameter $y$ ascends $V(x,y) = xy - 0.03\,y^2$. The equilibrium is the origin. Press Play and compare the two paths from the same start. Then raise $k$, which gives the critic more steps to catch up with its best response $y^* = x/0.06$ between generator steps.

The released code differs from Algorithm 1 as printed. In adversarial/__init__.py, the cost used by the MNIST config computes the discriminator’s and the generator’s gradients from the same minibatch and returns them together, so one SGD update moves both players at once. Figure 1’s caption in the paper also describes training as “simultaneously updating” D. With the non-saturating generator loss (next section) and a real network this worked; the toy game shows why simultaneous updates are not guaranteed to.

The swapped generator loss

Early in training the generator’s samples are poor, so the discriminator rejects them confidently and $D(G(\mathbf{z}))$ is near 0. The generator’s loss in (1) is $\log(1 - D(G(\mathbf{z})))$, and near $D = 0$ that function is nearly flat compared with the alternative below: its slope in $D$ is only $1/(1-D)$, about 1, and through the discriminator’s sigmoid the gradient reaching the generator shrinks in proportion to $D$ itself. The paper says “$\log(1 - D(G(\mathbf{z})))$ saturates” and proposes training $G$ to maximize $\log D(G(\mathbf{z}))$ instead:

The derivative of $-\log D$ has magnitude $1/D$. The ratio of the two is $(1-D)/D$. At $D = 0.01$ the old loss gives 1.01 and the new one 100, a factor of 99. The discriminator’s output is a sigmoid of a logit $\ell$, and following the chain rule one step further gives the same factor in a cleaner form: the gradient each fake passes back is proportional to $D$ under the old loss and to $1 - D$ under the new one. At $D = 0.003$ that is 0.003 against 0.997.

The paper states that the new objective “results in the same fixed point of the dynamics of G and D but provides much stronger gradients early in learning.” The two generator losses share that fixed point but differ at every other point, and with the swap the game is no longer one value function that one player maximizes and the other minimizes. The JSD result of the previous section was derived for the original loss. For the swapped loss, with the discriminator at its optimum, Arjovsky and Bottou (2017) showed the generator follows

The first term is the reverse KL divergence, $\mathbb{E}_{p_g}[\log(p_g/p_{\text{data}})]$. It is large when the generator puts samples where the data has almost no mass, and it changes little when the generator skips a region the data does cover, because that region contributes almost nothing to an expectation under $p_g$. The JSD enters with a minus sign. The trained loss is therefore not a descent on the JSD, and its reverse-KL term tolerates dropped modes, which the collapse section below returns to.

The listing puts the pieces together. It follows Algorithm 1’s structure with the non-saturating generator loss, which is what the paper recommends in §3 and what its code trains:

# one round of Algorithm 1: k discriminator steps, then one generator step
for _ in range(k):                         # the paper used k = 1
    x = sample_data(m)                     # m real examples,  x ~ p_data
    z = sample_noise(m)                    # m noise vectors,  z ~ p_z
    fake = G(z).detach()                   # no gradient reaches G on this line
    loss_d = -(log(D(x)) + log(1 - D(fake))).mean()
    opt_d.zero_grad(); loss_d.backward()
    opt_d.step()                           # minimizing -V = ascending V

z = sample_noise(m)
# Algorithm 1 as printed (saturating):   loss_g = log(1 - D(G(z))).mean()
loss_g = -log(D(G(z))).mean()              # non-saturating (paper, section 3)
opt_g.zero_grad(); loss_g.backward()       # gradient passes through D into G
opt_g.step()                               # only G's weights move here

Two lines control where gradients go. detach() in the discriminator step stops gradients from reaching $G$ there. In the generator step the gradient flows backward through $D$ into $G$; it also accumulates in D’s gradient buffers, which is harmless because only opt_g steps and opt_d.zero_grad() clears them before the next discriminator step. The generator never reads a real example: its only information about the data is the direction, in data space, that raises D’s score on its samples. For MNIST at batch size 100 the discriminator sees a 100×784 real batch and a 100×784 fake batch and outputs 100 probabilities for each.

A GAN trained in the page

Figure 6 is a real GAN, small enough to train in the browser in well under a second. It runs Algorithm 1 on 1-D data with two modes. The generator maps $z \sim U(-1, 1)$ through 16 tanh units to a number; the discriminator maps a number through 24 tanh units to a sigmoid. Both use SGD with momentum 0.5, batch 64. The generator starts as a narrow spike to the left of the data, and the discriminator gets a 300-step head start against it, the situation in the paper’s Figure 1(b). The violet curve is the trained discriminator, not $D^*$. The bottom readout, “push on each fake”, is the average size of the gradient each fake sample receives in data space.

At step 0 the discriminator scores the fakes at 0.0028, and the push on each fake is 2.71 under the non-saturating loss and 0.0077 under the saturating loss. Their ratio, 351, is $(1-D)/D$ at $D = 0.0028$, the factor derived above. Press Play with the non-saturating loss: within 10 generator steps the spike has moved onto the data; by step 1,500 the samples cover both modes and both traces in the lower strip sit near one half. Switch to the saturating loss and play again: for about 80 steps the spike creeps right while D(G(z)) stays under 0.01 and the push stays near 0.01. Once D(G(z)) starts to rise the push grows with it and the spike jumps onto the data. Raise $k$ to 5 and the stall lengthens to about 150 steps, because a discriminator trained more per round stays more confident.

Across six seeds, the non-saturating run reaches the data by step 10 in every case. The saturating run takes 80 to 170 steps at $k = 1$ and 110 to 290 at $k = 5$. Both losses end near the same place, which is what “same fixed point” predicts. The saturating run is slower because its generator gets almost no gradient while D rejects its samples.

Mode collapse and other failures

Seed 2 in Figure 6 shows the paper’s main named failure. The generator is graded only on whether $D$ accepts the samples it produces. If it finds an output that the current discriminator scores highly, it can map many values of $\mathbf{z}$ to that output, and nothing in its loss charges it for the modes it never visits: $D$ only scores the generator’s samples and real examples, and a mode the generator never visits produces no samples to score. The paper calls this “the Helvetica scenario”, in which $G$ “collapses too many values of $\mathbf{z}$ to the same value of $\mathbf{x}$ to have enough diversity to model $p_{\text{data}}$.” The later name is mode collapse. The paper lists it under keeping D and G synchronized: $G$ must not be trained too much without updating $D$.

A supervised loss does not allow this, because every training example has its own target and a missed example raises the loss. In a GAN, a discriminator that eventually learns to reject the one favored output pushes the generator to another single output, so the samples can hop from mode to mode instead of spreading across them. Figure 7 animates the two regimes on eight modes.

Mode collapse is one of three distinct failures, and each has a different cause. Oscillation comes from the game structure: gradient steps near a saddle point can circle it, as in Figure 4, and Mescheder, Geiger and Nowozin (2018) showed that when the data lies on a low-dimensional manifold, unregularized GAN training is not always convergent even locally. Vanishing generator gradients come from a discriminator that is too confident, through saturation (Figure 5) or through the flat region of Figure 3 where the two distributions overlap little. No single change to Algorithm 1 fixes all three, and much of the follow-up work targeted one of them.

The experiments and the metric

The paper trained adversarial nets on MNIST, the Toronto Face Database (TFD) and CIFAR-10. Generators used a mix of ReLU and sigmoid units. Discriminators used maxout units, each of which outputs the largest of several linear functions of its input, and were trained with dropout, which zeroes a random subset of units on every training step. In the MNIST config the discriminator has two maxout layers of 240 units with 5 pieces each, batch size is 100, the learning rate is 0.1, and momentum rises from 0.5 to 0.7.

Since a GAN has no likelihood to report, the paper estimated one. The procedure, a Gaussian Parzen window, places a Gaussian of width $\sigma$ on each generated sample, averages them into a density, and reports the mean log-likelihood of the test set under that density. $\sigma$ is chosen on a validation set. The paper says the method “has somewhat high variance and does not perform well in high dimensional spaces but it is the best method available to our knowledge.”

Figure 8 runs the estimator in one dimension on a generator that is perfect by construction: its samples are fresh draws from $p_{\text{data}}$, so the right answer, −1.17, is known. With 30 samples the best score over $\sigma$ is about −1.44. With 300 it reaches −1.17 at $\sigma$ near 0.1. At $\sigma = 0.01$ and 30 samples it falls to about −101, because most test points are far from every sample in units of $\sigma$. In 784 dimensions the number of samples needed to cover the space the same way is far larger, which is the “does not perform well in high dimensional spaces” caveat.

Table 1 reports these estimates for the adversarial net and three earlier models whose sampling runs a Markov chain: a deep belief network (DBN), a stacked contractive autoencoder (CAE) and a deep generative stochastic network (GSN). The adversarial net has the best MNIST score, 225 ± 2 against 214 ± 1.1 for the Deep GSN. On TFD it is second, 2057 ± 26 against 2110 ± 50 for the stacked contractive autoencoder, and those intervals overlap: 2110 − 50 = 2060 is below 2057 + 26 = 2083. The error bars mean different things on the two datasets, standard error across test examples on MNIST and across folds on TFD, with $\sigma$ chosen per fold.

The paper claims only that its samples are “at least competitive” and makes “no claim that these samples are better than samples generated by existing methods.” Theis, van den Oord and Bethge (2015) later showed that Parzen estimates can rank models differently from other likelihood estimates and concluded they “should generally be avoided.” The paper’s stronger evidence is qualitative. Its Figure 2 shows samples that are “fair random draws, not cherry-picked”, with the nearest training example beside each to show the model has not memorized the training set. Because sampling is one forward pass rather than a Markov chain, the samples are uncorrelated. Figure 3 of the paper shows digits from linear interpolation between points in $\mathbf{z}$ space.

Limits and what came after

The paper names two disadvantages. There is no explicit representation of $p_g(\mathbf{x})$, so a GAN cannot score how probable an image is. And $D$ must be kept synchronized with $G$ during training, the condition behind the Helvetica scenario. The advantages it lists are that Markov chains are never needed, only backpropagation is used to get gradients, no inference is needed during learning, and any differentiable function can serve as $G$. It also suggests a possible statistical advantage from the generator being updated only through the discriminator’s gradients, never directly with data examples, and separately notes that adversarial nets can represent “very sharp, even degenerate distributions,” where Markov-chain methods need blurry ones so their chains can mix between modes.

Section 7 lists extensions, and the first became its own paper within months: a conditional model $p(\mathbf{x} \mid \mathbf{c})$ obtained by feeding $\mathbf{c}$ to both G and D (Conditional GANs, November 2014). The others are an inference network that predicts $\mathbf{z}$ from $\mathbf{x}$, models of every conditional $p(\mathbf{x}_S \mid \mathbf{x}_{\bar S})$ with shared parameters, semi-supervised learning from discriminator features, and better ways to coordinate G and D or better distributions to sample $\mathbf{z}$ from during training.

DCGAN (November 2015) found convolutional architectures and training settings that made image GANs train reliably. Wasserstein GAN (January 2017) replaced the implicit JS objective with the Earth-Mover distance, whose gradient does not vanish when the two distributions barely overlap, which targets the flat region of Figure 3. Progressive growing (October 2017) and StyleGAN (December 2018) scaled the same game to high-resolution faces. From 2021, diffusion models overtook GANs on many image benchmarks; Diffusion Models Beat GANs on Image Synthesis reported better FID than the best GANs on ImageNet, with a single network trained on a denoising objective and no second player. GANs remain in use where one-pass sampling speed matters.