Classifier-Free Guidance (CFG): Classifier-Free Diffusion Guidance

Run a diffusion model with the label and without it, then amplify the difference.

Classifier guidance gave diffusion models a setting that trades sample variety for sample quality, but it needed a second network trained on noisy images. Classifier-free guidance gets the same setting from one network trained to work with and without its label, at the price of a second forward pass per sampling step.

On class-conditional ImageNet at 128×128 it reached an FID of 2.43, below the 2.97 of classifier guidance, and the same two-line change is the guidance_scale setting in Stable Diffusion and most text-to-image models since.

Some generative models let you trade variety for quality at sampling time, with no retraining. BigGAN draws its latent vector from a Gaussian and resamples any coordinate whose magnitude exceeds a threshold; a lower threshold gives more typical, cleaner images and fewer unusual ones. Glow scales the standard deviation of its prior by a temperature below one (0.7 in the Glow paper) for the same effect. Both work because a single noise vector is mapped straight to the output, so shrinking that vector moves every sample toward the center of what the model learned.

Diffusion models had no equivalent. Dhariwal and Nichol reported that the obvious analogues, scaling up the model's predicted scores or shrinking the noise added during sampling, give blurry, low-quality images (the classifier-free guidance paper cites this result; it does not re-run it). Their fix, in Diffusion Models Beat GANs on Image Synthesis, was classifier guidance: at every denoising step, add the gradient of a separate image classifier, so each sample is pushed toward images the classifier labels confidently. It worked, and it needed a classifier trained on half-noised images, which no standard pretrained classifier is.

Jonathan Ho and Tim Salimans (Google Research, 2022) remove the classifier. During training they replace the class label with a null token on 10 to 20 percent of examples, so one network learns to denoise both with and without the label. During sampling they run it both ways at every step and extrapolate from the label-free prediction past the labelled one. The guidance weight $w$ sets how far. On ImageNet 128×128 a weight of 0.3 cut FID from 7.27 to 2.43.

The sections below go in dependency order: what a diffusion model predicts, why that prediction is a gradient of log-density (a score), how classifier guidance adds a classifier's gradient to it, how Bayes' rule replaces that gradient with a difference of two predictions, how one network is trained to give both, what the guided sampler produces, and what the ImageNet sweep measured.

Trading diversity for fidelity at sampling time

Two properties of a set of samples pull against each other. Fidelity is how convincing and recognizable each sample is on its own. Diversity is how much of the real data's variety the set covers. Most of a model's probability mass sits on typical, unambiguous examples and the tails hold the unusual ones, so sampling closer to the high-density core raises fidelity and gives up the tails.

The two image metrics the paper reports measure these separately. Inception Score (IS) feeds each sample to a fixed ImageNet classifier and is high when the classifier is confident about each image and the predicted labels are spread across classes; it never looks at real images. Fréchet Inception Distance (FID) compares the distribution of classifier features over generated images with the distribution over real ones, and rises when the generated set loses variety or drifts away from real images. A BigGAN truncation sweep traces a curve: IS goes up and, past a point, FID goes up with it. The paper's goal is a diffusion sampler that produces the same kind of curve.

A diffusion model builds a sample through hundreds of small denoising steps, each adding fresh noise, so it has no single input vector to shrink; guidance changes the direction of every step instead.

A diffusion model in one screen

Start with a clean image $x$, every pixel scaled to roughly $[-1, 1]$. The forward process mixes in Gaussian noise. The paper uses the variance-preserving form, the one from DDPM and Variational Diffusion Models: the signal is shrunk by a factor $\alpha_\lambda$ while noise of scale $\sigma_\lambda$ is added.

Because $\alpha_\lambda^2 + \sigma_\lambda^2 = 1$ exactly, data with unit variance keeps unit variance at every noise level, hence the name. The variance-exploding convention used in score-based models keeps the signal at full scale and lets the noise variance grow without bound; formulas from one convention do not carry over to the other unchanged, because here the clean-image estimate has to divide by $\alpha_\lambda$.

The noise level is indexed by $\lambda = \log(\alpha_\lambda^2/\sigma_\lambda^2)$, the log signal-to-noise ratio. Large $\lambda$ is nearly clean and very negative $\lambda$ is nearly pure noise. The forward process runs toward decreasing $\lambda$, and sampling runs the other way, through an increasing sequence from $\lambda_{\min} = -20$ to $\lambda_{\max} = 20$. When the sampling loop below counts up, the image is getting cleaner. Two reference points: at $\lambda = 0$, $\alpha^2 = \sigma^2 = 0.5$; at $\lambda = -20$, $\alpha^2$ is about $2\times 10^{-9}$, so $z$ is the noise itself.

The network $\boldsymbol{\epsilon}_\theta$ is given $z_\lambda$ and $\lambda$ and predicts the noise $\boldsymbol{\epsilon}$ that was mixed in. Equation (1) is linear, so a noise estimate gives a clean-image estimate by solving for $x$:

For a worked pixel, set $\lambda = 0$, so $\alpha = \sigma = 0.707$, and take a clean value $x = 0.6$ and a noise draw $\epsilon = -1.2$: the noised value is −0.424 (0.424 − 0.849). If the network predicts $\hat\epsilon = -1.0$, off by 0.2, the recovered pixel is (−0.424 + 0.707) / 0.707 = 0.400, also off by 0.2. The noise error is multiplied by $\sigma/\alpha = e^{-\lambda/2}$, which is 1 at $\lambda = 0$ and 7.39 at $\lambda = -4$, where the same 0.2 noise error becomes a 1.48 pixel error. At high noise the best estimate of $x$ is an average over every clean image that could have produced $z$, which looks like a blur of many images.

Training adds known noise and penalizes the squared error of the noise prediction:

With a uniform $p(\lambda)$ this loss is proportional to the variational bound on log-likelihood, ignoring the decoder and prior terms. The schedule the paper uses is not uniform, which makes (5) a weighted bound whose weighting is chosen for sample quality. For conditional generation the network also receives the conditioning $c$ (a class label here, a text embedding in text-to-image models) and is written $\boldsymbol{\epsilon}_\theta(z_\lambda, c)$.

Noise prediction is the score

The score of a distribution is the gradient of its log-density, $\nabla_{z}\log p(z_\lambda)$: the direction in which a small move of $z$ makes it most probable under the noised data distribution. The optimal noise prediction is the score up to a minus sign and a factor of $\sigma_\lambda$:

The relation comes from Tweedie's formula. For a point corrupted by Gaussian noise, the least-squares best guess of the clean signal is the posterior mean, and Tweedie's formula writes it with the score:

A network trained with squared error learns that posterior mean, so its $x_\theta$ from the previous section equals $\mathbb{E}[x \mid z_\lambda]$. Setting the two expressions equal and cancelling $z$ gives the score relation:

The power of $\sigma$ is one in the score relation and two inside Tweedie's formula. As a check, let the clean data be a standard normal, $x \sim \mathcal{N}(0, 1)$. Then $z_\lambda$ is also standard normal at every $\lambda$, its score is $-z$, and the optimal noise prediction is $\sigma z$. At $\lambda = 0$ and $z = 1$: score $-1$, noise prediction 0.707, and posterior mean (1 − 0.5) / 0.707 = 0.707, which matches the textbook Gaussian answer $\alpha z$. The score points toward the data (from $z = 1$ back toward 0) and the predicted noise points the opposite way.

This identity turns operations on densities into arithmetic on network outputs. Multiplying densities adds their log-densities, which adds their scores, which adds the corresponding noise predictions after scaling by $-\sigma_\lambda$. Raising a density to a power multiplies its score by that power.

The relation is exact only for the true score. $\boldsymbol{\epsilon}_\theta$ is an unconstrained neural network, and nothing forces its output to be the gradient of any function: a gradient field has zero curl, and a network's output field can circulate. The section on the implicit classifier returns to this, because it decides what classifier-free guidance can be said to be.

How classifier guidance works

Classifier guidance takes a conditional diffusion model and a classifier $p_\theta(c \mid z_\lambda)$ trained to name the class of a noised image at any $\lambda$. At each sampling step it subtracts a multiple of the classifier's gradient from the noise prediction (subtracting from $\boldsymbol{\epsilon}$ adds to the score, since the two have opposite signs):

The bracket is the log of $p(z\mid c)\,p_\theta(c\mid z)^w$, so using $\tilde{\boldsymbol{\epsilon}}_\theta$ in place of $\boldsymbol{\epsilon}_\theta$ gives approximate samples from a reweighted distribution:

The power $w$ on a probability concentrates mass. Take two points that the conditional model finds equally likely. At one the classifier says $p(c\mid z) = 0.95$; at the other, in territory shared with another class, it says 0.5. At $w = 3$ the factors are $0.95^3 = 0.857$ and $0.5^3 = 0.125$, so the confident point gains weight relative to the contested one by a factor of 6.9 instead of 1.9. Raising $w$ thins each class out of the regions it shares with other classes and leaves it on its confident core.

Figure 1 is the paper's own toy (its Figure 2): three classes, each a Gaussian blob. At $w = 0$ you see the three plain conditionals overlapping. Move the slider up and watch the edges that face the other blobs fade while the far sides stay.

The tilt is applied at every noise level the sampler passes through, not only to clean images, and how much it moves the distribution depends on $\lambda$. Figure 2 takes a one-dimensional slice through two classes with clean data at $\pm 1$ (standard deviation 0.6) and draws the three factors separately: the noised class density, the classifier factor $p(c\mid z)^w$, and their normalized product. Set $w = 3$ and sweep $\lambda$. At $\lambda = 0$ the guided mean moves from 0.71 to 1.22 and the spread drops from 0.82 to 0.63. At $\lambda = 10$, where the classes barely overlap, the mean moves only 0.13. At $\lambda = -6$ both classes sit almost on top of each other at the origin, the classifier is close to 1/2 everywhere, and the spread is unchanged.

By raising $w$, Dhariwal and Nichol could move along an IS/FID curve the way BigGAN truncation does. Most of the cost is in the classifier: it has to read noisy images at every noise level, so a standard pretrained classifier cannot be used and a new one has to be trained for each dataset. Classifying the denoised estimate $x_\theta$ instead does not work either: at high noise that estimate is the blurred average described above, and the gradient would be taken through a different point than the one being sampled. And because the sampler follows a classifier's gradient toward higher confidence, the procedure can be read as a gradient-based adversarial attack on that classifier, which raises the question of whether improvements in classifier-based metrics like IS and FID reflect better images.

How classifier-free guidance works

The classifier enters the sampler only through its gradient $\nabla_{z}\log p(c\mid z_\lambda)$, so replacing that gradient replaces the classifier. Bayes' rule writes the classifier with generative quantities, $p(c\mid z) = p(z\mid c)\,p(c)/p(z)$. Taking the log and the gradient with respect to $z$, the prior $p(c)$ does not depend on $z$ and drops out:

The gradient of the classifier is the conditional score minus the unconditional score. Both are noise predictions, by the score relation. Write $\boldsymbol{\epsilon}^{*}$ for the exact scores (in noise units) of $p(z_\lambda\mid c)$ and $p(z_\lambda)$. The paper calls $p^{\,i}(c\mid z) \propto p(z\mid c)/p(z)$ the implicit classifier, and its gradient is

Substitute this into the classifier-guidance formula. The $-w\sigma_\lambda$ in front of the classifier gradient multiplies the $-1/\sigma_\lambda$ here, the $\sigma_\lambda$ factors cancel, and the two minus signs make a plus. With learned networks in place of the exact scores, the substitution gives the paper's Eq (6):

Both terms come from the same network, run with the label and with the null label. Eq (6) can be regrouped two ways:

The first form starts from the conditional prediction and adds $w$ more copies of the change the label made. The second starts from the unconditional prediction and moves $1+w$ times the distance to the conditional one, which for any $w > 0$ lands past the conditional prediction, on the far side from the unconditional one. Guidance extrapolates; it never blends toward the unconditional prediction. At $w = 0$ it is plain conditional sampling.

For one coordinate, say the conditional prediction is 0.20 and the unconditional one is 0.50, a difference of −0.30. At $w = 3$, Eq (6) gives 4(0.20) − 3(0.50) = −0.70. The first regrouping gives 0.20 + 3(−0.30) = −0.70 and the second 0.50 + 4(−0.30) = −0.70. The guided value lies 0.90 beyond the conditional prediction, on the side away from the unconditional one.

With exact scores, Eq (6) in score units is the gradient of $(1+w)\log p(z\mid c) - w\log p(z)$. The paper reads it this way in its discussion: guidance raises the conditional likelihood of the sample and lowers its unconditional likelihood, the second through a score term with a negative coefficient.

Figure 3 draws the three directions as vectors in the three-class toy, using exact scores. The probe is a noisy point you can drag. The gray arrow is the unconditional score, the teal arrow is the conditional score of the selected class, and the bright arrow is the guided combination. The amber dashed segment is the difference being amplified. Drag the probe into the region between two classes, where the two scores disagree most, and raise $w$ to watch the bright arrow extend past the teal one.

Training one network for both predictions

Eq (6) needs an unconditional prediction $\boldsymbol{\epsilon}_\theta(z_\lambda)$ next to the conditional one. Training a second, unconditional model would work but doubles the parameters and the training run. The paper trains one network for both: with probability $p_\text{uncond}$ each training example has its label replaced by a null token $\varnothing$, and the unconditional model is the same network given the null token:

The parameter count does not change, and the loss is still Eq (5). In code, $\varnothing$ is usually one extra row in the label-embedding table. DiT, a later class-conditional ImageNet model, uses an embedding table of 1,001 rows for 1,000 classes and sets the label to index 1000 with probability dropout_prob. In Stable Diffusion the null condition is the text encoder's embedding of the empty prompt. Figure 4 shows one training batch of 32 examples. Watch both ends of the slider: at 0 the network never sees $\varnothing$ and has no unconditional prediction to guide with, and at 1 it never sees a label.

One training step, with the shapes of the 64×64 ImageNet model (a $B$-image batch, labels in 0 to 999):

# Algorithm 1: one training step, batch of B images
x, c = sample_batch()                  # x: (B, 3, 64, 64), c: (B,) ints in 0..999
drop = rand(B) < p_uncond              # p_uncond = 0.1 or 0.2
c    = where(drop, NULL_ID, c)         # NULL_ID = 1000, an extra embedding row
lam  = sample_log_snr(B)               # one noise level per image, lambda ~ p(lambda)
eps  = randn_like(x)                   # the noise to be predicted
z    = alpha(lam) * x + sigma(lam) * eps
loss = mean((eps_theta(z, lam, c) - eps) ** 2)   # Eq (5), the same MSE for all
loss.backward(); opt.step()            # one set of weights learns c and NULL

The paper compared $p_\text{uncond}$ of 0.1, 0.2 and 0.5 on 64×64 ImageNet, all trained for 400,000 steps. 0.1 and 0.2 performed about the same across the whole IS/FID curve and 0.5 was worse everywhere (its best FID is 1.91, against 1.55 for 0.1). The authors conclude that only a small share of the model's capacity has to go to the unconditional task, and point out a parallel in classifier guidance, where small classifiers were enough.

The paper's discussion notes one case where the dropout can be skipped. With few classes and a known prior, the unconditional density is a mixture of the conditional ones,

so the unconditional score is the average of the conditional scores weighted by the posterior $p(c\mid z_\lambda)$. The toy figures on this page compute the unconditional score exactly that way. It costs one forward pass per class at each step, which the paper notes is inefficient for high-dimensional conditioning such as text.

What guided sampling produces

Sampling starts from pure noise at $\lambda_{\min} = -20$ and runs $T$ steps up to $\lambda_{\max} = 20$. Each step runs the network twice, forms Eq (6), converts the result to a clean estimate, and takes one step of the sampler:

# Algorithm 2: sampling with guidance weight w
z = randn(B, 3, 64, 64)                       # z_1 ~ N(0, I) at lambda_min = -20
for t in range(T):                            # lambda increases: noisy -> clean
    lam = lams[t]
    e_c = eps_theta(z, lam, c)                # conditional pass
    e_u = eps_theta(z, lam, NULL_ID)          # unconditional pass
    e   = (1 + w) * e_c - w * e_u             # Eq (6)
    x   = (z - sigma(lam) * e) / alpha(lam)   # implied clean image
    if t == T - 1:
        return x
    z = ancestral_step(z, x, lam, lams[t + 1], v)  # Eq (4); DDIM also works

The paper's sampler is ancestral: each step draws the next $z$ from a Gaussian centered on a blend of the current $z$ and the clean estimate, with fresh noise whose variance interpolates (in log space, with a coefficient $v$) between two standard choices. The 64×64 models use $v = 0.3$ and the 128×128 models $v = 0.2$. Algorithm 2 notes that another sampler can be substituted, for example the deterministic DDIM update, which moves to the next noise level keeping the predicted noise fixed, with no fresh noise.

The classifier-guidance derivation suggests the samples come from $p(x\mid c)\,p(c\mid x)^w$, the tilt of Figure 1 applied to clean images. The paper calls them approximate samples, and Chidambaram et al. (2024) prove that guided sampling does not produce that distribution. The guidance is applied at every noise level, including the intermediate ones (around $\lambda = 0$ in Figure 2) where the noised classes overlap and the tilt moves the most mass. Those pushes happen before a sample has settled into its class, and the sampler carries them through to the clean sample.

Figure 5 runs the sampler on the three-class toy with exact scores, 140 samples per class and 64 steps from $\lambda = -20$ to 20. Press Play to watch the cloud of noise split into three, or drag the step slider. At the last step, the white plus marks the mean of the exactly tilted distribution and the white ring marks the mean of the guided samples. At $w = 0$ the samples reproduce the data: the mean stays put and the spread matches the data's 0.80. At $w = 3$ under DDIM, each class's samples move 1.41 outward, away from the other two classes, and their spread falls to 0.54, while the tilted target moves only 0.06 because these clean classes barely overlap. Switch to the ancestral sampler and the outward shift at $w = 3$ drops to 0.46 with spread 0.68: the fresh noise late in sampling lets samples relax back toward the class, so how far guidance pushes depends on the sampler as well as on $w$.

In this toy, large weights put samples outside the region where the class has most of its data, beyond the side of the class that faces away from the others. The paper reports that strongly guided ImageNet samples show saturated colors; one reading, consistent with the toy but not tested in the paper, is that saturation is the image-space version of samples pushed past the typical range of their class.

Why there is no classifier inside

Eq (6) was derived from the implicit classifier, so it is natural to describe classifier-free guidance as classifier guidance with a classifier computed by Bayes' rule. The paper does not, because the derivation used the exact $\boldsymbol{\epsilon}^{*}$ and the sampler uses the learned $\boldsymbol{\epsilon}_\theta$.

With exact scores, the guided prediction $(1+w)\boldsymbol{\epsilon}^{*}(z,c) - w\boldsymbol{\epsilon}^{*}(z)$ is the classifier-guided prediction for $p^{\,i}$. The deployed sampler uses two outputs of an unconstrained network. Their difference $\boldsymbol{\epsilon}_\theta(z,c) - \boldsymbol{\epsilon}_\theta(z)$ is not constrained to have zero curl, so in general it is not the gradient of any scalar function, and no classifier $p(c\mid z)$ exists whose guidance reproduces it. Two consequences follow, both stated in the paper. Classifier-free guidance resembles classifier guidance without being an instance of it. And since no classifier gradient appears anywhere in Eq (6), a step along it cannot be read as a gradient-based adversarial attack on an image classifier, so its gains in IS and FID come from a pure generative model.

The paper treats the Bayes derivation as informal. It cites Grandvalet and Bengio (2004), who found that discriminative classifiers generally outperform classifiers derived from generative models, and notes that under a misspecified model a classifier derived by Bayes' rule can be inconsistent.

The guidance scale: w versus s

The paper's weight $w$ is 0 for no guidance. Most code writes the same formula from the unconditional side, the second regrouping above, with its own scale $s$:

The diffusers library writes exactly this line, turns guidance off when guidance_scale is 1 or less, and defaults to 7.5 in its Stable Diffusion pipeline. The Imagen paper writes $w\boldsymbol{\epsilon}(z,c) + (1-w)\boldsymbol{\epsilon}(z)$ and says that $w = 1$ disables guidance, so Imagen's $w$ is this $s$ under a different letter. Conversions: paper $w = 0.3$ is $s = 1.3$; paper $w = 3$ is $s = 4$; Stable Diffusion's 7.5 is paper $w = 6.5$, beyond the largest weight the paper measured (4.0).

A different "+1" appears inside classifier guidance and should not be confused with this one. Guiding an unconditional model with weight $w+1$ targets the same distribution as guiding a conditional model with weight $w$:

This holds because $p(z\mid c) \propto p(z)\,p(c\mid z)$. Dhariwal and Nichol got their best results guiding an already-conditional model, and this paper stays in that setup. The classifier scale in their ADM paper is a third, unrelated number; their tuned classifier scales should not be read as values of $w$.

Results on ImageNet

The experiments use class-conditional ImageNet at 64×64 and 128×128, with the model architectures and hyperparameters of Dhariwal and Nichol's guided diffusion models (tuned for classifier guidance, the authors note, so possibly not ideal here). The 64×64 models trained for 400,000 steps and the 128×128 models for 2.7 million. The guidance weight was swept over $w \in \{0, 0.1, \dots, 1.0, 2, 3, 4\}$ and each setting scored with FID and IS on 50,000 samples.

The two metrics, as defined in the papers the authors cite: IS uses the Inception network's class probabilities $p(y\mid x)$ on each sample and their average $p(y)$ over samples:

It is high when each image gets a confident label and the labels vary across images, and it uses no real data. FID fits a Gaussian to the 2,048-dimensional Inception features of real images (mean $\mu_r$, covariance $\Sigma_r$) and of generated ones ($\mu_g, \Sigma_g$):

The trace term grows when the generated features are less spread out than the real ones, so a model that collapses onto a few confident outputs is penalized even if each output is a good image.

Figure 6 plots the paper's sweeps (Table 1 at $p_\text{uncond} = 0.1$, Table 2 at $T = 256$), IS on the horizontal axis and FID on the vertical. On 64×64, IS rises at every step of the sweep, from 53.71 unguided to 260.2 at $w = 4$. FID does not move in one direction: it falls from 1.80 at $w = 0$ to its minimum of 1.55 at $w = 0.1$, then rises to 26.22 at $w = 4$. The 128×128 sweep (at $T = 256$ steps) has the same shape: FID 7.27 unguided, 2.43 at $w = 0.3$, 21.53 at $w = 4$, while IS climbs from 82.45 to 421.03. The paper's text describes FID as monotonically decreasing in $w$; its tables show a U-shape, and the Provenance panel below records the discrepancy.

A small amount of guidance improves both metrics at once. On 128×128, going from $w = 0$ to $w = 0.3$ cuts FID by a factor of three (7.27 to 2.43) and nearly doubles IS (82.45 to 158.47). Past the minimum, each increase in IS costs FID. The best-FID weight depends on the setup: 0.1 on 64×64, 0.3 on 128×128 at $T = 256$ and at $T = 1024$ (both 2.43), but 0.4 at $T = 128$ (3.02, against 3.04 at 0.3).

Against other models on 128×128, the FID of 2.43 is below classifier-guided ADM-G (2.97), LOGAN (3.36), CDM (3.52) and BigGAN-deep (5.7), and the paper calls its 128×128 results state of the art in the literature at the time. At the other end of the sweep, $w = 4$ gives FID 21.53 and IS 421.03, better on both metrics than BigGAN-deep at its maximum-IS truncation (FID 25, IS 253). The 64×64 result does not lead: its best FID of 1.55 is behind CDM's 1.48, though ahead of ADM's 2.07.

Costs and limitations

Each sampling step evaluates the network twice. The paper compares its $T = 256$ results with ADM-G because ADM-G uses about the same number of steps, but ADM-G's second evaluation per step is a classifier that is typically smaller than the diffusion model. With the same architecture as ADM-G, the comparison at equal compute is the paper's $T = 128$ setting, and there the best FID is 3.02, worse than ADM-G's 2.97. Removing the classifier simplifies training, and each sampling step costs more than it would with a small classifier. The authors suggest injecting the conditioning late in the network, so that most of the computation could be shared between the two passes, and leave it to future work. In practice the two passes are run as one batch of twice the size (diffusers and DiT both do this), which uses the hardware better but performs the same arithmetic.

Guidance also reduces diversity, by design. The paper's discussion points out that in deployed models this can hurt the parts of the data distribution that are already underrepresented, and lists raising sample quality without reducing diversity as an open problem. On ImageNet the paper's metrics put the best trade at the FID minimum, $w = 0.1$ to $0.3$. Stable Diffusion's default of $s = 7.5$ (paper $w = 6.5$) is far past it, and the paper's experiments do not measure what such weights buy in a text-to-image model.