Diffusion Models Beat GANs on Image Synthesis

A classifier's gradient steers a diffusion model past the best GAN.

The gap to GANs closes in two moves: a better denoising network, then classifier guidance, a single dial that trades diversity for fidelity by nudging each sampling step along a classifier's gradient.

In 2021 GANs held the image-generation record and everyone knew their weakness: they drop whole regions of the data. This paper closes the quality gap without inheriting that weakness.

By 2021 the sharpest synthetic images came from GANs, generative adversarial networks: a generator and a discriminator locked in a forgery game until the fakes pass. On ImageNet, BigGAN-deep held the best $\text{FID}$ (the standard image-quality score, defined below). But GANs pay for that fidelity in two ways. They are delicate to train, collapsing without carefully chosen regularizers, and they cover less of the data than the best likelihood-based models: they tend to lock onto a handful of convincing modes and skip the rest.

Diffusion models sit on the other side of that trade. They are stable to train, they scale predictably, and they cover the distribution well, but in 2021 their samples still looked a notch softer than a good GAN's on hard datasets like ImageNet. Dhariwal and Nichol set out to close that gap from both ends. Their diagnosis had two parts: the diffusion architecture had simply been explored less than the GAN one, and diffusion models had no knob for trading coverage for per-sample sharpness the way a GAN's truncation trick does.

So the paper is two contributions bolted together. First, a careful architecture sweep produces a stronger denoising network, which the authors call ADM (their ablated diffusion model). Second, classifier guidance: a way to steer sampling toward a class using the gradients of an image classifier, with one scalar dial that trades diversity for fidelity. Together they reach an FID of $2.97$ on ImageNet 128×128, $4.59$ on 256×256, and $7.72$ on 512×512, past BigGAN-deep on every one, while keeping more of the distribution. And with a fast sampler the guided model matches BigGAN using as few as $25$ forward passes.

The plan below: build the diffusion model just enough to see where a class label and a classifier can enter, fix the architecture, pin down what "fidelity" and "diversity" actually measure, then derive classifier guidance and watch the trade-off it buys.

A diffusion model in one page

This is not the paper's contribution, so it goes fast; for the full construction see the DDPM explainer this one leans on. A diffusion model turns noise into an image by reversing a fixed noising process. The forward process takes a clean image $x_0$ and, over many steps, mixes in Gaussian noise until nothing is left but static. In the variance-preserving convention this paper uses, you can jump straight to any noise level $t$ with one formula:

Here $\bar\alpha_t$ slides from near $1$ (clean) down to near $0$ (pure noise) as $t$ grows, and $\epsilon \sim \mathcal{N}(0,\mathbf{I})$ is the noise. Two things move at once: the signal is scaled down by $\sqrt{\bar\alpha_t}$ while noise of standard deviation $\sqrt{1-\bar\alpha_t}$ is added. Because the two variances sum to one for unit-scaled data, the total variance stays pinned near one throughout, which is what "variance-preserving" names. (That contrasts with the variance-exploding convention some diffusion papers use, where the signal is left untouched and only the noise grows. Same family, different bookkeeping, and the algebra below is specific to this one.)

To generate, you learn to walk the process backward. The move that makes it tractable, from Ho et al., is to train a network $\epsilon_\theta(x_t,t)$ to predict the noise $\epsilon$ that was added, not the clean image. That gives a plain regression target: noise an image yourself so you know the answer, then penalize the squared error.

Once you can predict the noise, algebra recovers a slightly cleaner image and you repeat. Written out, the reverse step is a Gaussian whose mean comes straight from the noise prediction (with $1-\alpha_t = \beta_t$):

The paper inherits two upgrades from Improved DDPM (Nichol & Dhariwal's companion work). Instead of fixing the reverse-step variance, it is learned as a per-coordinate interpolation, in log space, between the two natural bounds $\beta_t$ and $\tilde\beta_t$, with a network output $v$:

Because the two bounds differ by orders of magnitude near $t \to 0$, interpolating in log space keeps that blend well-conditioned. Learning the variance this way buys graceful degradation when you take fewer, larger sampling steps, so the paper's 25-step guided sampling stays accurate later. When it wants even fewer steps, the paper switches to the deterministic DDIM sampler.

The last identity in this section carries the guidance argument that follows, so I will state it on its own. Because the noised image in (1) is a Gaussian around $\sqrt{\bar\alpha_t}\,x_0$, the gradient of its log-density, the score, equals the predicted noise rescaled and flipped in sign:

Read (5) with two things in mind. You divide by the standard deviation $\sqrt{1-\bar\alpha_t}$, not the variance $1-\bar\alpha_t$: the Gaussian score carries a factor $(x_t-\sqrt{\bar\alpha_t}x_0)/(1-\bar\alpha_t)$, and the numerator is itself $\sqrt{1-\bar\alpha_t}\,\epsilon$, cancelling one power of the root. The minus sign follows from the geometry: the predicted noise points away from the data, so density increases against it. This equivalence, that a noise predictor is a score estimator, is the same one the score-matching line of work rests on, made precise by Song et al.. Through it, a classifier's gradient can enter the sampler.

A better denoising network

The first half of the paper is unglamorous and effective: take the U-Net that diffusion models had been using and sweep its knobs on ImageNet 128×128, keeping what lowers FID. Each change is borrowed from elsewhere in the literature. Every later experiment in the paper uses the network this sweep produces.

The changes they test, each applied on its own to the old baseline: making the network deeper rather than wider at a fixed size; using more attention heads (a head is one self-attention pass, where every spatial location reads from every other; more heads run more passes in parallel, each on its own slice of the channels); adding self-attention at the 32, 16, and 8 resolutions instead of only at 16×16; swapping in BigGAN's residual blocks (layers whose output is added back onto their input) for up- and down-sampling; and rescaling every residual branch, the added-back path, by $1/\sqrt{2}$. Drag through them below. Four help; the last one hurts and is dropped, and depth helps FID but trains too slowly to keep. The kept three, stacked, beat the sum of their parts.

The attention sweep hides a second lever. Beyond raising the head count from one to four, the authors also try fixing the channels per head instead. Fewer channels per head means more heads at a given width, and the sweep shows that helps: 32 channels per head gives the best FID, but 64 is chosen as the wall-clock compromise, which also matches how a Transformer usually sizes its heads. So raising the head count and lowering the channels per head adjust the same quantity.

Injecting the timestep and class

Every residual block has to be told which noise level it is working at, and, for a class-conditional model, which class. The old way added a projection of that information to the activations. The paper's block instead uses adaptive group normalization (AdaGN): first group-normalize the activations (split the channels into a few groups and rescale each group to zero mean and unit variance, from the activations' own statistics), then scale and shift them per channel using values read from the timestep-and-class embedding.

Only the GroupNorm uses statistics of the activations $h$. The scale $y_s$ and shift $y_b$ are a learned linear projection of the embedding, applied after normalization. (In an unconditional model there is no class term, so $y$ is only the timestep's projection.) This is the feature-wise modulation of FiLM (Perez et al. 2018) and the adaptive instance norm of StyleGAN, applied to a diffusion U-Net. The reason it beats plain addition is that addition can only move a normalized channel, while a scale-and-shift can also stretch or compress it, a strictly richer control. Toggle between the two below and watch the widths: under addition every channel keeps the reference spread.

With those choices settled, ADM already sets a new state of the art on the unconditional LSUN datasets and on ImageNet 64×64. On harder, higher-resolution ImageNet it is close to BigGAN-deep but not past it. Closing that last gap is the job of the second half of the paper.

Measuring fidelity and diversity

The paper's argument is a trade between two things, so both need definitions before the trade makes sense. Sample quality has more than one axis, and the metrics capture different ones.

FID (Fréchet Inception Distance) is the default overall score. Push both the real images and the generated ones through an Inception-V3 network, fit a Gaussian to each cloud of feature vectors, and measure the distance between those two Gaussians:

Lower is better. Because it compares whole distributions, FID penalizes a model that drops coverage (its $\Sigma_g$ shrinks) as well as one that makes ugly samples. Be precise about what it is, though: FID is the squared 2-Wasserstein distance (the least total work to slide one Gaussian's probability mass onto the other's) between Gaussian fits of the features, not between the true (non-Gaussian) feature distributions, so it inherits that modelling error. A spatial variant, sFID, uses spatial feature maps instead of pooled ones to catch structure the pooled statistics average away.

Inception Score rewards samples that a classifier labels confidently and that spread across many classes. Its blind spot is coverage within a class: a model that memorizes one perfect example per class scores near the top. So the paper leans on a pair of metrics that separate the two axes cleanly. Precision measures fidelity as the fraction of generated samples that land inside the real-data manifold, the region of feature space real images actually occupy. Recall measures diversity as the fraction of real data that lands inside the generated manifold. Both manifolds are estimated the nonparametric way, as the union of small nearest-neighbour spheres around each point.

The two directions are not interchangeable: precision tests your samples against the real manifold, recall tests the real data against your manifold, two different sets and not one counted twice. Drag the concentration knob below: as the model's samples pull onto one dominant mode, every sample stays inside the real manifold (precision holds high) while whole modes of real data go uncovered (recall falls). A truncated GAN does that collapse to gain fidelity, and classifier guidance is about to do it on purpose.

The reference point for that trade is BigGAN's truncation trick: sample the GAN's latent from a normal distribution truncated toward its centre. Truncating harder concentrates on the modes, which raises fidelity and drops diversity, the dial diffusion models were missing. The rest of the paper builds that dial.

Guiding with a classifier: the exact part

Suppose you have a trained unconditional diffusion model and you want class-$y$ images out of it, without retraining. GANs bake in class information through class-conditional statistics; the diffusion problem is getting a label into a model that never saw one. The paper's answer is to bring in a separate classifier $p_\phi(y\mid x_t)$ and let its gradients pull sampling toward the class.

Part of this is correct with no approximation. One later step is not exact, so keep the two apart. To condition each reverse step on $y$, it suffices to multiply the model's step by the classifier and renormalize:

with $Z$ a normalizing constant. The appendix proves the companion fact this rests on: adding a label to the forward noising process changes nothing about it, so the unconditional reverse process the model already learned is still the right thing to multiply. Both of those are exact. The trouble is only that sampling from a product of a Gaussian and an arbitrary classifier is intractable, so the next step approximates it. This product-of-distributions form of guiding a diffusion step goes back to Sohl-Dickstein et al. (2015); the score version comes from Song et al. (2021).

The mean shift

The mean shift below rests on one approximation, and its accuracy sets the limits of everything after. The model's reverse step is a Gaussian $\mathcal{N}(\mu, \Sigma)$. The classifier $\log p_\phi(y\mid x_t)$ is some curved function of $x_t$; over the small region the Gaussian covers, replace it by its first-order Taylor expansion around the mean $\mu$, a straight-line stand-in for the curved classifier that is trustworthy only close to $\mu$. A linear term added to a Gaussian's log-density just moves its mean, and completing the square shows the move is by $\Sigma g$:

where $g = \nabla_{x_t}\log p_\phi(y\mid x_t)\big|_{x_t=\mu}$ is the classifier gradient at the mean. The covariance is unchanged; only the mean slides, and it slides by $\Sigma g$, not by the raw gradient $g$. The $\Sigma$ in front rescales the shift by the step's own uncertainty: since the reverse-step covariance is roughly $\tilde\beta_t\mathbf{I}$, the shift is large when the step is uncertain and small once the sample is nearly settled. The sampler adds one scalar dial, the gradient scale $s$, in front of the shift:

Drag $\mu$ and the scale $s$ in the figure. The gradient $g$ points from wherever you are toward the target class, the shift carries the sampling mean that way, and the readout shows the classifier's probability of the target climbing as $s$ grows. Put the point out near the other class, where an unconditional sample would be ambiguous, and watch guidance rescue it toward the target.

The approximation in (9) is exact only in the limit of many, tiny steps, where the reverse-step covariance $\Sigma$ shrinks toward zero and the classifier looks locally linear. The paper says as much: the Taylor expansion is justified "in the limit of infinite diffusion steps, where $\lVert\Sigma\rVert \to 0$." That gap is why the fast, few-step sampler later needs a larger, hand-tuned scale to offset the coarser steps.

Turning up the gradient

The scale $s$ should logically be $1$: that is the value the derivation produces for genuine conditional sampling. But the authors found that on unconditional ImageNet, $s=1$ barely worked. At scale $1$ the classifier reported only about 50% confidence in the intended class on the finished samples, and the images did not look like the class. Pushing the scale up fixed it: the class probabilities climbed toward 100%, and the samples snapped into the class. The paper's corgi figure reports it on the unconditional model: classifier scale $1.0$ gives FID $33.0$ (worse than the unguided $26.21$) and unconvincing dogs, while scale $10.0$ gives FID $12.0$ and clear corgis.

There is a clean reason a bigger scale sharpens rather than distorts. Scaling the gradient by $s$ is the gradient of a classifier raised to the power $s$:

Raising a probability distribution to a power $s>1$ and renormalizing is temperature sharpening at $T = 1/s$: it amplifies the peaks, so guidance concentrates harder on the classifier's modes and produces higher-fidelity, less diverse samples. Read (11) precisely, though. The identity treats $Z$ as a constant, which is the gradient of an unnormalized energy; a genuinely per-input normalized $p^s$ would add a $-\nabla_x\log Z(x)$ term. So $p^s$ is the right intuition, a temperature knob, not a literal claim about sampling from a renormalized classifier.

Amplifying a classifier's gradient by ten is also the standard recipe for an adversarial example, a tiny gradient step that fools a classifier into full confidence on an image that looks like noise. So why does guidance produce corgis instead of adversarial static? The relevant fact from the paper is that the classifier is trained on noised images, the same noising the diffusion model uses. A classifier that has only ever seen clean images has brittle, high-frequency input gradients, the ones adversarial attacks exploit. Training on noise forces it to rely on features robust to noise, so its gradients point toward class-meaningful structure a human would recognize. (The paper states the noised-classifier training and the outcome; reading it through the adversarial-robustness lens is the interpretation offered here, supported by later work on perceptually-aligned gradients, not a claim the paper makes.)

A naming caution, because a sibling paper reuses the letter differently. This paper's $s$ is a raw multiplier on a separate classifier's gradient. Classifier-free guidance, the follow-up that removes the separate classifier entirely, uses a guidance weight $w$ with its own scale $s_{\text{cfg}} = w+1$. They are different parameterizations of related ideas; do not read one paper's number into the other.

Guiding the fast sampler

The mean-shift in (10) is a statement about the stochastic reverse step, so it cannot be dropped into a deterministic sampler like DDIM, which has no per-step covariance to shift. Identity (5) does the work here. The joint distribution's score is the model's score plus the classifier's gradient, because the log of a product is a sum:

Match that back to a noise prediction using (5), and you get a single guided noise $\hat\epsilon$ you can feed into ordinary DDIM in place of $\epsilon_\theta$:

The sign flips here. Adding the classifier gradient to the score means subtracting it from the noise, because the score is minus the noise over $\sqrt{1-\bar\alpha_t}$ by (5). Plug (13) back into (5) and the joint score comes out exactly, so this is the same guidance, rewritten for a sampler that speaks in $\epsilon$ rather than in a mean and covariance. Two algorithms, one idea:

# classifier-guided sampling (stochastic), gradient scale s
x = randn(shape)                       # start: x_T ~ N(0, I)
for t in reversed(range(T)):           # T .. 1
    mu, Sigma = model(x, t)            # unconditional reverse step
    g = grad_x(log p_phi(y, x, t))     # classifier gradient at x
    x = sample_normal(mu + s * Sigma @ g, Sigma)
return x                               # x_0

# classifier-guided DDIM (deterministic), gradient scale s
x = randn(shape)
for t in reversed(range(T)):
    eps = eps_theta(x, t)              # predicted noise
    g   = grad_x(log p_phi(y, x, t))   # classifier gradient
    eps_hat = eps - s * sqrt(1 - abar[t]) * g
    x = ddim_step(x, eps_hat, t)       # ordinary DDIM, using eps_hat
return x

The classifier is cheap next to the generator: only the U-Net's downsampling trunk with an attention pooling head, trained on the same noised images. So guidance can ride the 25-step DDIM sampler and still match a GAN.

The trade-off, and beating BigGAN

Now the dial has real numbers on it. Sweeping the gradient scale traces a curve in the precision/recall plane. The figure plots the paper's measured points on ImageNet 256×256: raise $s$ and each point climbs (more precision, more fidelity) and slides left (less recall, less coverage). FID, annotated at each point, is lowest not at the extreme but in the middle.

Read the conditional curve. With no guidance the model has FID $10.94$, precision $0.69$, recall $0.63$. At $s=1$ it hits FID $4.59$, precision $0.82$, recall $0.52$: sharper samples, some coverage given up, and the best FID on the sweep. Push to $s=10$ and precision climbs to $0.88$ while recall collapses to $0.32$ and FID worsens to $9.11$. FID is U-shaped because it responds to both axes: too little fidelity or too little diversity both raise it, so its minimum sits at an intermediate scale. That is why the paper tunes the scale per dataset rather than cranking it, using values from $0.5$ up to $9.0$ across resolutions and samplers.

Set against BigGAN-deep, the numbers make the paper's case. The GAN reaches precision $0.87$ at recall $0.28$, deep in the corner where fidelity is bought by abandoning coverage. Guided diffusion reaches comparable precision while keeping much more recall, and it wins outright on FID: $2.97$ on ImageNet 128, $4.59$ on 256, $7.72$ on 512, each past the corresponding BigGAN-deep number. When guidance is combined with a second, upsampling diffusion stage (ADM-U), FID improves further to $3.94$ on 256 and $3.85$ on 512. And because the learned variance tolerates coarse sampling, the guided model matches BigGAN using as few as 25 forward passes, cutting deep into the speed advantage that was diffusion's last real disadvantage.

The recipe reads as two independent wins that happen to combine. A denoising network tuned like a modern vision model closes most of the quality gap on its own. A classifier's gradient, folded into each sampling step, supplies the fidelity dial GANs had and diffusion models lacked, and it does so without a second adversarial network to train or a whole conditional model to retrain. Within a year the field had taken the next steps: classifier-free guidance removed the separate classifier, and guided diffusion became the backbone of text-to-image systems like Stable Diffusion.