VerifiedarXiv:2112.1074127 min
Diffusion · Text-to-image · Image editing

GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models

A diffusion model draws images from captions, follows the caption more closely when each denoising step pushes its prediction past the captioned prediction, away from the caption-free one, and can be fine-tuned to repaint erased regions.

GLIDE (OpenAI, 2021) trained a 3.5-billion-parameter diffusion model on captioned images and compared two ways to push its samples toward the caption: the gradient of a CLIP model, and classifier-free guidance. Human raters preferred classifier-free guidance on both photorealism and caption match, and preferred its samples to DALL-E's 87% of the time on photorealism.

Explaining the paperGLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion ModelsNichol, Dhariwal, Ramesh, Shyam, Mishkin, McGrew, Sutskever, Chen · OpenAI · ICML 2022 · arXiv:2112.10741 ↗

Below: the diffusion model and the two places the caption enters it, both guidance methods derived and then run side by side on a toy with exact scores, the Elo arithmetic behind the human evaluations, the inpainting fine-tune, and the step counts that make one image take 15 seconds.

What GLIDE is

GLIDE (Guided Language to Image Diffusion for Generation and Editing) produces a 256×256 image from a caption in two stages. A 3.5-billion-parameter diffusion model generates a 64×64 image conditioned on the caption. A 1.5-billion-parameter diffusion model, also conditioned on the caption, upsamples it to 256×256. Both are pixel-space diffusion models built on the U-Net from Diffusion Models Beat GANs (the ADM architecture). Each also contains a Transformer that encodes the caption and is trained together with the U-Net.

The paper asks one main question: what is the best way to make such a model follow its caption? It tests two methods that strengthen the effect of the caption at sampling time. CLIP guidance nudges each denoising step along the gradient of a CLIP model's image-caption similarity. Classifier-free guidance runs the model with and without the caption and extrapolates from the second prediction past the first. Human raters preferred classifier-free guidance, so every final GLIDE sample uses it.

Two more results come out of the same model. Fine-tuned with four extra input channels, it fills an erased region of a real image to match a caption and the surrounding pixels, which makes it a text-driven image editor. And a 300M-parameter version trained on filtered data, GLIDE (filtered), was released with code; the 3.5B model was not.

The diffusion model underneath

GLIDE uses the Gaussian diffusion model of DDPM. A clean image x0x_0, with pixel values scaled to [−1,1][-1, 1], is corrupted over T=1000T = 1000 steps. Each step shrinks the image slightly and adds Gaussian noise:

q(xt∣xt−1)=N(xt; αt xt−1, (1−αt) I)q(x_t \mid x_{t-1}) = \mathcal{N}\big(x_t;\ \sqrt{\alpha_t}\,x_{t-1},\ (1-\alpha_t)\,I\big)q(xt∣xt−1)=N(xt; αt xt−1,(1−αt) I)\begin{aligned} q(x_t \mid x_{t-1}) = \mathcal{N}\big(&x_t;\ \sqrt{\alpha_t}\,x_{t-1}, \\ &(1-\alpha_t)\,I\big) \end{aligned}
(1)

Composing the steps gives the noised image at any step directly. With αˉt=α1α2⋯αt\bar\alpha_t = \alpha_1 \alpha_2 \cdots \alpha_t,

xt=αˉt x0+1−αˉt ϵϵ∼N(0,I)x_t = \sqrt{\bar\alpha_t}\, x_0 + \sqrt{1 - \bar\alpha_t}\, \epsilon \qquad \epsilon \sim \mathcal{N}(0, I)xt=αˉt x0+1−αˉt ϵϵ∼N(0,I)\begin{gathered} x_t = \sqrt{\bar\alpha_t}\, x_0 + \sqrt{1 - \bar\alpha_t}\, \epsilon \\ \epsilon \sim \mathcal{N}(0, I) \end{gathered}

The released code uses the cosine schedule for the base model, under which αˉ500=0.494\bar\alpha_{500} = 0.494. Halfway through, the image and the noise carry equal weight, 0.7030.703 and 0.7110.711: a pixel with value 0.6 and a noise draw of −1.2 becomes 0.703 × 0.6 + 0.711 × (−1.2) = −0.43. By t=1000t = 1000 the image term is gone and xTx_T is standard Gaussian noise.

Generation runs the chain backward. When each forward step adds little noise, the reverse step q(xt−1∣xt)q(x_{t-1} \mid x_t) is close to a Gaussian, so the model predicts one:

pθ(xt−1∣xt)=N(μθ(xt), Σθ(xt))p_\theta(x_{t-1} \mid x_t) = \mathcal{N}\big(\mu_\theta(x_t),\ \Sigma_\theta(x_t)\big)
(2)

Sampling starts from xT∼N(0,I)x_T \sim \mathcal{N}(0, I) and draws xT−1,xT−2,…,x0x_{T-1}, x_{T-2}, \dots, x_0 one step at a time. The network does not output the mean directly. It is trained to predict the noise that was mixed in, with a squared error:

Lsimple=Et, x0, ϵ[ ∥ϵ−ϵθ(xt,t)∥2 ]L_{\text{simple}} = \mathbb{E}_{t,\, x_0,\, \epsilon}\big[\,\|\epsilon - \epsilon_\theta(x_t, t)\|^2\,\big]
(3)

A noise prediction determines everything else. Solving the mixing equation for the image gives the model's current guess of the clean image, x^0=(xt−1−αˉt ϵθ)/αˉt\hat x_0 = (x_t - \sqrt{1-\bar\alpha_t}\,\epsilon_\theta)/\sqrt{\bar\alpha_t}, and μθ\mu_\theta is the mean of the reverse step toward that guess. The noise prediction is also a rescaled score, the gradient of the log density of noised images:

∇xtlog⁡p(xt)=−ϵ∗(xt)1−αˉt\nabla_{x_t} \log p(x_t) = -\frac{\epsilon^*(x_t)}{\sqrt{1 - \bar\alpha_t}}
(4)

ϵ∗\epsilon^* is the exact noise prediction that ϵθ\epsilon_\theta approximates. The minus sign sets the direction used in guidance: the predicted noise points away from likely images, so moving xtx_t against it moves the sample toward them.

The variance Σθ\Sigma_\theta is learned rather than fixed, following Nichol and Dhariwal's improved DDPM. The network outputs six channels per pixel: three for ϵθ\epsilon_\theta and three values vv that interpolate, in log space, between the two natural choices of variance, the forward step's βt=1−αt\beta_t = 1 - \alpha_t and the posterior variance β~t\tilde\beta_t. A learned variance lets the model sample well with far fewer steps than it was trained on, which GLIDE relies on: it trains on 1000 steps and samples with 27 to 250 of them.

How the caption reaches the U-Net

The caption is tokenized into a sequence of KK tokens (128 slots in the released code, padded) and encoded by a Transformer that is part of the diffusion model and trained with it. In the 3.5B model the Transformer has 24 residual blocks of width 2048, about 1.2B parameters; the image U-Net is the ADM ImageNet 64×64 architecture widened to 512 base channels, about 2.3B.

The Transformer's output reaches the U-Net in two ways. First, one vector summarises the caption. The final token embedding is projected and used where ADM puts its class embedding: it is added to the timestep embedding, and the sum sets a per-channel scale and shift in every residual block. Second, the whole sequence of token embeddings is projected separately for each attention layer of the U-Net and concatenated to that layer's attention context. In the code the projected tokens become extra keys and values placed in front of the image's own, so every pixel position attends over all text tokens and all other positions in one softmax.

The shapes, for the released 300M model (192 base channels, a 16-layer Transformer of width 512) whose configuration is public:

# Text path of the released GLIDE base model (text2im_model.py), shapes for one
# caption. Released config: text_ctx 128, xf_width 512, 16 layers, 192 channels.
tokens, mask = bpe(caption), pad to 128 with end_token   # (128,), mask (128,)
h = token_emb(tokens) + pos_emb                          # (128, 512)
h = where(mask, h, padding_emb)                          # learned emb for pads
xf_out = layer_norm(transformer(h))                      # (128, 512), no causal mask

pooled = Linear(512, 4 * 192)(xf_out[-1])                # (768,) last slot
emb = time_embed(t) + pooled                             # drives every res block

# inside every attention layer of the U-Net, at a level with C channels:
img = norm(feature_map).reshape(C, H * W)                # one column per pixel
q, k, v = qkv(img).split(3)                              # each (C, H*W)
tk, tv = Conv1d(512, 2 * C, 1)(xf_out.T).split(2)        # each (C, 128)
k = concat([tk, k], axis=-1)                             # (C, 128 + H*W)
v = concat([tv, v], axis=-1)
out = softmax(q.T @ k / sqrt(d_head)) @ v.T              # per head: text + pixels

The U-Net has attention at 32×32, 16×16 and 8×8. At 16×16, each of the 256 positions attends over 128 text keys plus 256 image keys, 384 in all; at 8×8 the text keys are two thirds of the 192. In the released code the pooled vector is the Transformer output at the last of the 128 slots. For any caption shorter than 128 tokens that slot holds a padding token, which still sees the whole caption because the text Transformer has no causal mask.

Why a text model needs guidance

A diffusion model trained on (image, caption) pairs models p(x∣c)p(x \mid c). Sampling from it faithfully reproduces the training distribution, including its looseness: web captions often describe only part of their image, so the images that go with any one caption vary widely, and so do the samples. In the paper's human evaluation the guided 3.5B model beats the unguided one by 156.9 Elo points on photorealism (Table 4, below). Guidance sharpens the conditional distribution at sampling time: it gives up some sample diversity to get images that match their caption more closely.

The template is classifier guidance from Dhariwal and Nichol. Train a classifier pϕ(y∣xt)p_\phi(y \mid x_t) on noised images, and at each reverse step shift the mean of (2) along the gradient of its log probability for the target label:

μ^θ(xt∣y)=μθ(xt∣y)+s Σθ(xt∣y) ∇xtlog⁡pϕ(y∣xt)\hat\mu_\theta(x_t \mid y) = \mu_\theta(x_t \mid y) + s\,\Sigma_\theta(x_t \mid y)\,\nabla_{x_t} \log p_\phi(y \mid x_t)μ^θ(xt∣y)=μθ(xt∣y)+s Σθ(xt∣y) ∇xtlog⁡pϕ(y∣xt)\begin{aligned} \hat\mu_\theta(x_t \mid y) &= \mu_\theta(x_t \mid y) \\ &+ s\,\Sigma_\theta(x_t \mid y)\,\nabla_{x_t} \log p_\phi(y \mid x_t) \end{aligned}
(5)

The shift is the gradient times the step's own variance, so it is large early in sampling, when the step is uncertain, and small near the end. Applied to an unconditional model with s=1s = 1, the shift approximately samples p(x∣y)p(x \mid y); larger ss raises the classifier's probability to the power ss, which concentrates samples on images the classifier is most confident about. A classifier over 1,000 ImageNet labels is easy to train. Free-form captions have no fixed label set to train such a classifier over; the next two sections cover GLIDE's two replacements for one.

CLIP guidance with a noised CLIP

CLIP has an image encoder f(x)f(x) and a caption encoder g(c)g(c), trained so that the dot product f(x)⋅g(c)f(x) \cdot g(c) is high for matching pairs and low for mismatched ones. That dot product scores how well an image matches any caption, so it can stand in for the classifier's log probability in (5):

μ^θ(xt∣c)=μθ(xt∣c)+s Σθ(xt∣c) ∇xt(f(xt)⋅g(c))\hat\mu_\theta(x_t \mid c) = \mu_\theta(x_t \mid c) + s\,\Sigma_\theta(x_t \mid c)\,\nabla_{x_t}\big(f(x_t) \cdot g(c)\big)μ^θ(xt∣c)=μθ(xt∣c)+s Σθ(xt∣c) ∇xt(f(xt)⋅g(c))\begin{aligned} \hat\mu_\theta(x_t \mid c) &= \mu_\theta(x_t \mid c) \\ &+ s\,\Sigma_\theta(x_t \mid c)\,\nabla_{x_t}\big(f(x_t) \cdot g(c)\big) \end{aligned}
(6)

Two details make (6) work. The model being guided is already text-conditional, so CLIP guidance adds to the caption's effect rather than supplying all of it. And the CLIP model must be trained on noised images. The public CLIP models only ever saw clean photos, so f(xt)f(x_t) for a half-noise xtx_t is an input far outside their training data and its gradient is unreliable. Earlier CLIP-guided diffusion notebooks compensated with image augmentations and extra perceptual losses. GLIDE trains a noised CLIP instead: an image encoder f(xt,t)f(x_t, t) that takes the noised image and the timestep, trained with CLIP's contrastive objective on images noised with the base model's schedule.

The noised CLIP used in the paper is a ViT-L at 64×64 with 4×4 patches, so each image becomes 16 × 16 = 256 patches. It was trained for 390K iterations at batch size 32K on an even mix of the CLIP and DALL-E datasets, then fine-tuned for 30K iterations on a broader set of internet images. In the released code the guiding gradient is that of the scaled cosine similarity: both embeddings are normalised to unit length and their dot product is multiplied by CLIP's learned logit scale before differentiating. Every guided step therefore costs a forward and a backward pass through the CLIP image encoder on top of the diffusion model's forward pass.

How classifier-free guidance works in GLIDE

Classifier-free guidance, from Ho and Salimans, removes the separate model. One diffusion network is trained to make both a conditional prediction ϵθ(xt∣c)\epsilon_\theta(x_t \mid c) and an unconditional one ϵθ(xt∣∅)\epsilon_\theta(x_t \mid \varnothing), and the sampler combines them:

ϵ^θ(xt∣c)=ϵθ(xt∣∅)+s (ϵθ(xt∣c)−ϵθ(xt∣∅))\hat\epsilon_\theta(x_t \mid c) = \epsilon_\theta(x_t \mid \varnothing) + s\,\big(\epsilon_\theta(x_t \mid c) - \epsilon_\theta(x_t \mid \varnothing)\big)ϵ^θ(xt∣c)=ϵθ(xt∣∅)+s (ϵθ(xt∣c)−ϵθ(xt∣∅))\begin{aligned} \hat\epsilon_\theta(x_t \mid c) &= \epsilon_\theta(x_t \mid \varnothing) \\ &+ s\,\big(\epsilon_\theta(x_t \mid c) - \epsilon_\theta(x_t \mid \varnothing)\big) \end{aligned}
(7)

At s=1s = 1 this returns the conditional prediction. For s>1s > 1 it starts from the unconditional prediction and moves ss times the distance to the conditional one, past the conditional prediction itself. Ho and Salimans write the same rule with a weight w=s−1w = s - 1, so GLIDE's s=3s = 3 is their w=2w = 2.

The "unconditional" prediction comes from the same network with an empty caption. After the main training run, GLIDE fine-tuned the base model with 20% of the token sequences replaced by the empty sequence ∅\varnothing (in the released code, 128 padding tokens with an all-false mask); the model keeps its text-conditional skill and also learns to generate with no caption.

The form of (7) comes from an implicit classifier. Bayes' rule gives p(c∣xt)∝p(xt∣c)/p(xt)p(c \mid x_t) \propto p(x_t \mid c) / p(x_t), so the gradient of the log-classifier is the difference of two scores, and (4) turns each score into a noise prediction:

∇xtlog⁡p(c∣xt)=∇xtlog⁡p(xt∣c)−∇xtlog⁡p(xt)=−ϵ∗(xt∣c)−ϵ∗(xt)1−αˉt\nabla_{x_t} \log p(c \mid x_t) = \nabla_{x_t} \log p(x_t \mid c) - \nabla_{x_t} \log p(x_t) = -\frac{\epsilon^*(x_t \mid c) - \epsilon^*(x_t)}{\sqrt{1-\bar\alpha_t}}∇xtlog⁡p(c∣xt)=∇xtlog⁡p(xt∣c)−∇xtlog⁡p(xt)=−ϵ∗(xt∣c)−ϵ∗(xt)1−αˉt\begin{aligned} \nabla_{x_t} \log p(c \mid x_t) &= \nabla_{x_t} \log p(x_t \mid c) - \nabla_{x_t} \log p(x_t) \\ &= -\frac{\epsilon^*(x_t \mid c) - \epsilon^*(x_t)}{\sqrt{1-\bar\alpha_t}} \end{aligned}
(8)

Now guide the unconditional score with this classifier at scale ss: ∇log⁡p(xt)+s ∇log⁡p(c∣xt)\nabla \log p(x_t) + s\,\nabla \log p(c \mid x_t). Multiplying through by −1−αˉt-\sqrt{1-\bar\alpha_t} turns the sum back into noise predictions and gives exactly (7), with exact predictions in place of learned ones. The derivation explains the form; it does not make classifier-free guidance a classifier method. The learned difference ϵθ(xt∣c)−ϵθ(xt∣∅)\epsilon_\theta(x_t \mid c) - \epsilon_\theta(x_t \mid \varnothing) is not guaranteed to be the gradient of any function, and the guided sampler does not draw from p(xt) p(c∣xt)sp(x_t)\,p(c \mid x_t)^s exactly.

For one pixel at t=500t = 500, take xt=−0.43x_t = -0.43 (the pixel from earlier) and say the conditional model predicts noise 0.30 there and the unconditional model 0.10. With s=3s = 3 the guided prediction is 0.10 + 3 × 0.20 = 0.70. The implied clean pixel (xt−0.711 ϵ^)/0.703(x_t - 0.711\,\hat\epsilon)/0.703 is −0.92 for the conditional prediction, −0.72 for the unconditional one, and −1.32 for the guided one: guidance pushed the pixel past the conditional value and outside the valid range of [−1,1][-1, 1]. The released sampler clips the predicted clean image to [−1,1][-1, 1] at every step, which keeps such overshoots from compounding. (Imagen later replaced that clip with dynamic thresholding to allow much larger scales.)

The guided step in the released notebook:

# One guided step, as in the released notebook (text2im.ipynb). The batch holds
# the same x_t twice: first half with the caption, second half with empty tokens.
def model_fn(x_t, t, tokens, mask):
    half = x_t[: len(x_t) // 2]
    out = model(concat([half, half]), t, tokens=tokens, mask=mask)  # 6 channels
    eps, rest = out[:, :3], out[:, 3:]          # rest: learned variance values
    cond_eps, uncond_eps = split(eps, 2)
    half_eps = uncond_eps + s * (cond_eps - uncond_eps)   # s = 3.0 in the notebook
    eps = concat([half_eps, half_eps])
    return concat([eps, rest], dim=1)          # variance taken unguided

# p_sample_loop then turns (eps, rest) into the mean and variance of
# p(x_{t-1} | x_t), clips the implied x_0 to [-1, 1], and draws x_{t-1}.

Guidance touches only the three noise channels. The learned-variance channels pass through, so the step's variance is the conditional model's. The paper gives two reasons to prefer this method over CLIP guidance: the model guides itself with its own knowledge of images and captions rather than with a separate, possibly smaller model, and nothing has to be trained to classify captions, which have no fixed set of labels.

The two methods on a toy

Figure 1 runs both methods on a model small enough to compute exactly. The data are three Gaussian clusters, A, B and C, in two dimensions. The caption is "A", and the caption-conditional model is deliberately weak: it puts half its mass on A (with a narrower spread than the data) and a quarter on each of B and C, the way a model trained on loose captions does. The unconditional model is the plain three-cluster mixture. Every ϵ∗\epsilon^* is computed in closed form, and 320 samples are drawn with 100 DDPM steps on the cosine schedule. For CLIP guidance the toy uses a linear scorer, f(x)=xf(x) = x and g(c)g(c) a unit vector pointing at A, and reports the same g⋅xg \cdot x as its "CLIP score".

Figure 1 · classifier-free guidance vs a CLIP gradient
3.0
Switch the method and drag the scale. s = 1 (classifier-free) and s = 0 (CLIP gradient) are the same unguided conditional sampler. Teal: samples in the caption cluster; grey: in B or C; violet: farther than 2.5 standard deviations from every cluster (dashed rings). The toy CLIP score is the mean of g · x.

Without guidance, 54% of the samples land in A. Classifier-free guidance moves samples into A as ss grows and shrinks their spread inside it: 80% at s=3s = 3, 99% at s=10s = 10, where the spread has fallen from 0.40 to 0.14. That lost variation is the diversity cost the paper measures as falling recall. The CLIP gradient first does the same job faster, with 95% in A at s=3s = 3 and the spread untouched. Past that, the constant push along gg carries samples out of A's ring toward the upper left, where no data lives; at s=10s = 10 56% of samples are off the data. The toy CLIP score rises the whole time, to 2.65 against a ceiling of about 1.7 for classifier-free guidance (the score of A's center), because the scorer rewards distance along gg whether or not the point looks like data.

The paper saw the same split at scale. It sampled MS-COCO captions at 64×64 and swept both scales, scoring the samples four ways. FID measures how far the statistics of generated images are from those of real images (lower is better); Inception Score rewards images that an ImageNet classifier labels confidently and with varied labels; precision is the share of samples that look like real data, and recall the share of real data the samples cover. On the FID-against-Inception-Score and precision-against-recall curves, classifier-free guidance is nearly Pareto optimal: no CLIP-guidance setting beats it on both axes at once. Plotted as CLIP score against FID, the order reverses: CLIP guidance raises CLIP score far more. The authors' hypothesis is the one the toy acts out, that CLIP guidance finds adversarial examples for the evaluation CLIP model, images that score well without matching the caption better. The human evaluations in the next section test it.

How people judged the samples

Each human comparison shows a rater two 256×256 images from different systems and asks one question: which looks more photorealistic, or which matches the caption better. The rater can declare neither better, which counts as half a win for each. A photorealism evaluation collects 1,000 comparisons; caption evaluations also collect 1,000, except the guidance-scale sweeps, which use 500. Wins are converted into Elo scores, the rating system from chess.

Under the Elo model, a system with score σi\sigma_i beats one with score σj\sigma_j with probability

P(i beats j)=11+10(σj−σi)/400P(i \text{ beats } j) = \frac{1}{1 + 10^{(\sigma_j - \sigma_i)/400}}
(9)

A 400-point gap means 10:1 odds. Scores are fit by maximum likelihood: with AijA_{ij} the number of times system ii beat system jj, starting every σi\sigma_i at 0, minimise

Lelo=−∑i,jAijlog⁡11+10(σj−σi)/400L_{\text{elo}} = -\sum_{i,j} A_{ij} \log \frac{1}{1 + 10^{(\sigma_j - \sigma_i)/400}}
(10)

The paper prints this objective with σi−σj\sigma_i - \sigma_j in the exponent. Read literally, that is the log probability that jj beats ii, and minimising it rewards losing: the fit returns the correct scores with their signs flipped. The published tables follow the standard form (10), since the system raters preferred has the highest score. Only differences of σ\sigma enter (9), so the scores are meaningful only relative to the other systems in the same table.

Figure 2 · Elo scores and what a gap means
compare
Tap two bars (or the buttons) to compare them: the curve converts their gap into the predicted share of comparisons the first one wins. "Objective as printed" shows the fit the paper's formula would give for the same wins.

Table 1 pits the two methods against each other and against no guidance, each at its best scale from a preliminary sweep. Classifier-free guidance scores 82.7 on photorealism and 110.9 on caption similarity; CLIP guidance −73.2 and 29.3; unguided sampling −88.6 and −106.2. A gap of 155.9 points means raters would pick the classifier-free sample over the CLIP-guided one 71.0% of the time on photorealism; the caption gap of 81.6 points means 61.5%. CLIP guidance beats no guidance clearly on caption match (68.6%) but barely on photorealism (52.2%), which fits the hypothesis: it moved images toward the caption as CLIP sees it, without making them look more real.

Results against DALL-E and on COCO

Against DALL-E, OpenAI's earlier text-to-image model, a 12-billion-parameter autoregressive model trained with roughly the same compute, GLIDE wins every comparison in Table 3. The paper ran three settings, each at two DALL-E sampling temperatures (1.0 and 0.85):

SettingDALL-E temp.GLIDE wins: photorealismcaption
no reranking1.091%83%
no reranking0.8584%80%
DALL-E reranked with CLIP1.089%71%
DALL-E reranked with CLIP0.8587%69%
reranked, GLIDE passed through DALL-E's VAE1.072%63%
reranked, GLIDE passed through DALL-E's VAE0.8566%61%

CLIP reranking means drawing many DALL-E samples and keeping the ones CLIP scores highest, which multiplies DALL-E's sampling cost; GLIDE uses no reranking. The last two rows push GLIDE's images through DALL-E's discrete VAE, which blurs them the way DALL-E's own outputs are blurred, to separate image sharpness from content. GLIDE still wins 66% to 72% of photorealism comparisons, so most of its lead survives the blurring. The abstract's headline figures, 87% and 69%, are the reranked row at temperature 0.85.

On MS-COCO, without ever training on it, GLIDE reaches a zero-shot FID of 12.24 (30,000 samples, reference images center-cropped). DALL-E's zero-shot FID is about 28 and LAFITE's 26.94. Models trained on COCO itself score better (lower): the GANs XMC-GAN (9.33) and LAFITE (8.12). Removing validation images that resemble training images, as the DALL-E paper did, shrinks the reference set by 21% and raises GLIDE's FID to 12.89, a change the authors attribute mostly to FID's bias on smaller reference sets.

Table 4 compares guidance with scale. Moving from the 300M model to the 3.5B model (about 12 times the parameters; the paper says roughly 10), both unguided, gains 107.9 Elo on photorealism and 109.3 on caption match. Adding classifier-free guidance to the 300M model gains 160.0 and 207.3. The guided 300M model beats the unguided 3.5B model, 57.4% of photorealism comparisons and 63.7% on caption match. Select Table 4 in Figure 2 to read these pairs.

How GLIDE edits images by inpainting

Inpainting fills a masked region of an existing image. Any diffusion model can do it without retraining by replacement: sample as usual, and after each step overwrite the known pixels with a fresh noised copy of the original, xtknown∼q(xt∣x0)x_t^{\text{known}} \sim q(x_t \mid x_0). The paper found that this occasionally leaves edge artifacts, and names the reason: the model never sees the context itself, only a noised version of it. Early in sampling the known pixels are almost pure noise, so the content generated in the hole is barely constrained by its surroundings, and late in sampling, when the context is clear, the hole's content is already decided.

GLIDE fine-tunes for the task instead. During fine-tuning, random regions of training images are erased, and the model receives the remaining pixels and a mask as extra inputs. The first convolution gets four new input channels, a second RGB image and a mask channel, whose weights are initialised to zero; at the start of fine-tuning the model computes exactly what the pretrained model did, and it learns from there how to use the context. The upsampler is fine-tuned the same way, and it always receives the full low-resolution image but only the unmasked part of the high-resolution one.

# Inpainting fine-tune (Sec. 4.3) with the input layout of InpaintText2ImUNet.
# mask = 1 where pixels are kept, 0 where the model must generate.
def train_step(x0, caption):                     # x0: (3, 64, 64) in [-1, 1]
    mask = random_erase_mask()                   # (1, 64, 64)
    t = randint(1, 1000); eps = randn_like(x0)
    x_t = sqrt(abar[t]) * x0 + sqrt(1 - abar[t]) * eps
    inp = concat([x_t, x0 * mask, mask])         # (7, 64, 64): 3 + 4 new channels
    loss = mse(model(inp, t, caption)[:3], eps)  # + the learned-variance term

# The first conv gets 4 extra input channels whose weights start at zero, so at
# step 0 the fine-tuned model computes exactly what the pretrained one did.
# Sampling (inpaint.ipynb) also overwrites the known pixels of each predicted x_0:
#   x0_hat = x0_hat * (1 - mask) + image * mask

Figure 3 shows why the clean context matters, on a 48-pixel "image" whose prior is a Gaussian process (a smooth random curve), so that both methods can use the exact score. The top panel is replacement with the exact unconditional model. The bottom panel is the ideal fine-tuned model, which samples the hole from the exact conditional distribution given the clean context; GLIDE's fine-tuned network is a learned approximation of it. Play the sampler to see what each model receives as context at each step.

Figure 3 · replacement vs a model that sees the context
step 100
14 px
steps
Play or scrub the sampling steps; change the hole width and the number of steps. Amber: the context as the model receives it (noised in the top panel, clean in the bottom). Teal: six samples. Dashed: the true signal that was erased.

At the default 14-pixel hole, the six replacement samples fan out across the hole (spread 0.77 at its center) while the conditional samples follow the erased signal (spread 0.04). Scrub back to step 35 to see why: the top panel's context is still mostly noise when the hole's coarse shape is being decided. In 2,000-sample runs of the same toy (a 14-pixel hole, 100 steps), the spread at the center was 0.94 for replacement, close to the prior's 1.0, against 0.07 for the conditional model, and the seams jumped 0.20 against 0.12. Raising the step count to 1000 does not close the gap. A 4-pixel hole is easy for both, and a 40-pixel hole leaves even the exact conditional model unsure about the middle, because the context says little about pixels far from it.

The released inpainting notebook adds one more constraint at sampling time: after each step it writes the known pixels into the model's predicted clean image, so the prediction always agrees with the context exactly. With the fine-tuned model, the paper shows edits that insert objects with matching shadows and reflections, edits to paintings that follow the painting's style, and a complex scene built by generating an image and then applying a series of inpainting edits. It also combines inpainting with SDEdit: the user paints a rough colour sketch into the masked region, the image is noised to an intermediate step, and the model denoises it into a realistic version of the sketch.

The fine-tuned model sometimes ignores the caption and fills the hole with whatever the surroundings suggest. In Appendix E the authors counter this by masking out the context channels entirely, which reduces the model to the replacement method: it then follows the caption more consistently, but its insertions fit the scene less smoothly.

What one image costs

The paper's figures sample the base model with 150 steps (100 for inpainting); its evaluations use 250 steps, which gives slightly better FID. With classifier-free guidance every step evaluates the 3.5B model twice, so a 250-step sample is 500 evaluations of the base model, run as 250 forward passes on a doubled batch.

The upsampler uses a hand-tuned 27-step schedule, found by sweeping FID. The 1000 training steps are split into five ranges of 200, and the sampler takes 10, 10, 3, 2 and 2 evenly spaced steps in them, from the least noisy range to the noisiest. The released schedule is

0, 22, 44, 66, 88, 111, 133, 155, 177, 199, 200, 222, 244, 266, 288, 311, 333, 355, 377, 399, 400, 500, 599, 600, 799, 800, 997

Twenty of the 27 steps are spent below step 400, where the upsampler adds fine detail, and two at 800 or above. The code swaps the last step, 999, for 997, with the comment that this reduces artifacts from the noisiest timesteps. The released notebook runs the upsampler with the deterministic DDIM sampler and scales the starting noise by 0.997. The unoptimised total is 15 seconds per image on one A100 GPU, against a single forward pass for a GAN.

Training used DALL-E's dataset and roughly DALL-E's compute. The base model trained for 2.5M iterations at batch size 2048, the upsampler for 1.6M iterations at batch size 512, both in 16-bit precision with standard loss scaling (as in mixed precision training).

The released model: GLIDE (filtered)

The authors judged the 3.5B model too capable of producing convincing disinformation to release. They instead trained a 300M-parameter model on a filtered dataset of several hundred million internet images, mostly disjoint from the CLIP and DALL-E data. Three filters removed images of people, of violent objects, and of two hate symbols. Each filter is an SVM with an RBF kernel on CLIP ViT-B/16 features, its bias tuned for less than 1% false negatives; on a held-out batch of 1,024 images, neither the people filter nor the violence filter let through a single false negative. GLIDE (filtered) was pretrained for 1.1M iterations and fine-tuned for 500K more for classifier-free guidance and inpainting; a filtered upsampler (192 base channels, text width 512) trained for 400K iterations. A noised ViT-B CLIP trained on filtered data was released alongside.

Red-teaming with adversarial prompts found no recognisable images of people. The filtered model still carries dataset biases, which classifier-free guidance amplifies: "toys for girls" yields more pink toys and stuffed animals than "toys for boys", and "a religious place" tends toward church-like buildings. A regurgitation test sampled 30,000 training prompts and inspected the generated images closest to their training images in CLIP space; none reproduced its training image faithfully.

Limitations

GLIDE fails on prompts that describe unusual objects or scenes; the paper's examples are "an illustration of a cat that has eight legs", "a bicycle that has continuous tracks instead of wheels", "a mouse hunting a lion" and "a car with triangular wheels". The 300M models, trained on either the full or the filtered data, bind attributes to the wrong objects more often than the 3.5B model and do worse on compositions such as stacked coloured cubes. Sampling is slow, at 15 seconds an image. The CLIP-guidance comparison rests on one noised CLIP model, and the adversarial-example explanation for its high CLIP scores is a hypothesis: the human ratings support it, and the paper does not look for adversarial structure in the images directly.

GLIDE's design was reused soon after. DALL-E 2's decoder is a modified GLIDE model conditioned on CLIP image embeddings, and classifier-free guidance, which GLIDE showed working on free-form text, became the default for text-to-image models including Imagen and latent diffusion.

Provenance Verified against primary literatureHow we verify
Nichol et al. (2021), Sec. 2.1q(x_t|x_{t-1}) = N(sqrt(alpha_t) x_{t-1}, (1 - alpha_t) I); p_theta(x_{t-1}|x_t) = N(mu_theta(x_t), Sigma_theta(x_t)); L_simple = E ||eps - eps_theta(x_t, t)||^2. Sigma_theta is learned with the method of Nichol & Dhariwal (2021).
Nichol et al. (2021), Sec. 2.2 to 2.4Classifier guidance mu_hat = mu + s Sigma grad log p_phi(y|x_t); classifier-free eps_hat = eps(x_t|0) + s (eps(x_t|c) - eps(x_t|0)) with s >= 1; CLIP guidance mu_hat = mu(x_t|c) + s Sigma grad (f(x_t) . g(c)).
Nichol et al. (2021), Sec. 4.1, 4.23.5B base model at 64x64: ADM ImageNet-64 architecture at width 512 (about 2.3B) plus a text Transformer of 24 residual blocks at width 2048 (about 1.2B). 1.5B upsampler to 256x256 with base width 384 and text width 1024. Base: 2.5M iterations at batch 2048; upsampler: 1.6M at batch 512. Fine-tuning for guidance replaces 20% of captions with the empty sequence.
glide-text2im, text2im_model.py, get_text_emb and forwardThe pooled text vector is transformer_proj(xf_out[:, -1]), the last of 128 padded positions, added to the timestep embedding. Padded positions get a learned padding_embedding; the text Transformer (xf.py) uses no attention mask.
glide-text2im, unet.py, AttentionBlock and QKVAttentionencoder_kv = conv_nd(1, encoder_channels, channels * 2, 1) projects the 128 token embeddings to keys and values, which are concatenated in front of the image keys and values (k = cat([ek, k]), v = cat([ev, v])) before one softmax.
glide-text2im, notebooks/text2im.ipynbmodel_fn applies guidance to the 3 eps channels only and passes the variance channels through; guidance_scale = 3.0, 100 respaced base steps in the notebook, upsampler with timestep_respacing fast27 via ddim_sample_loop and noise scaled by 0.997.
glide-text2im, clip/model_creation.py, cond_fnThe CLIP gradient is grad_x of exp(logit_scale) times the dot product of L2-normalised image and text embeddings; the image encoder also receives t. gaussian_diffusion.py condition_mean: new_mean = mean + variance * gradient.
glide-text2im, respace.py, space_timestepsfast27 = sections 10, 10, 3, 2, 2 over five 200-step ranges of the 1000-step process, then step 999 is swapped for 997 ("help reduce DDIM artifacts from noisiest timesteps").
glide-text2im, text2im_model.py InpaintText2ImUNet, notebooks/inpaint.ipynbInput = cat([x, inpaint_image * inpaint_mask, inpaint_mask]), 3 + 3 + 1 = 7 channels; mask = 1 on kept pixels. The notebook uses guidance_scale = 5.0 and a denoised_fn that writes the known pixels into every predicted x_0.
Nichol et al. (2021), App. A, BHuman evaluation: 1,000 pairwise comparisons per photorealism eval, 1,000 per caption eval except 500 in guidance-scale sweeps; ties count half a win. Noised CLIP: ViT with 4x4 patches at 64x64, 390K iterations at batch 32K. Sampling: 150 base steps for figures, 250 for evaluations, 100 for inpainting.
Elo rating, standard formExpected score E_A = 1 / (1 + 10^((R_B - R_A)/400)), so a 400-point gap means 10:1 odds.
Ho & Salimans (2022), Eq. (6)Writes guidance as (1 + w) eps(z, c) - w eps(z); GLIDE's s equals that w plus 1, so GLIDE s = 3 is w = 2 there.
correctionTwo slips in the paper's equations, one place where the code differs from the text, and one hidden sign. (1) The Elo objective in App. A.1, -sum A_ij log(1/(1 + 10^((sigma_i - sigma_j)/400))) with A_ij the number of times model i beats model j, uses the probability that j beats i. Minimising it as printed gives every model the negated score of the correct fit, so the most-preferred model would rank last; the tables (classifier-free guidance at +82.7 and +110.9) are consistent with the standard form 10^((sigma_j - sigma_i)/400), which this page uses. (2) Sec. 2.3 writes the gradient of the implicit classifier as grad log p^i(x_t|y); the implicit classifier defined one line earlier is p^i(y|x_t). Its "proportional to eps*(x_t|y) - eps*(x_t)" also hides a negative constant: the score is -eps*/sqrt(1 - alpha_bar_t), so the classifier gradient points along eps*(x_t) - eps*(x_t|y). (3) Sec. 2.4 writes the CLIP guidance target as the raw dot product f(x_t) . g(c). The released code differentiates exp(logit_scale) times the dot product of L2-normalised embeddings, a cosine similarity, and passes t to the image encoder; the gradient of a cosine has no component along the image embedding, so the two do not steer identically.

Questions you might still have

?

Is GLIDE the model inside DALL-E 2?
Its architecture is. The DALL-E 2 (unCLIP) paper builds its image decoder by modifying the GLIDE architecture: it adds the CLIP image embedding to the timestep embedding and as four extra context tokens, and keeps the text path. The DALL-E 2 explainer covers that decoder. GLIDE itself conditions only on the caption and has no prior over CLIP embeddings.

?

Why did humans prefer classifier-free guidance when CLIP guidance scored higher on CLIP score?
The paper's hypothesis is that CLIP guidance finds adversarial examples: it changes pixels in whatever way raises the guiding CLIP model's similarity score, and some of those changes raise the evaluation CLIP model's score too without making the image match the caption better to a person. The evaluation used the public CLIP ViT-B/16, a different network from the noised ViT-L used for guidance, so the effect has to transfer between models. The human evaluation is the test of the hypothesis; the paper does not analyse the guided images for adversarial structure directly.

?

Does classifier-free guidance need a second model?
No. One network learns both predictions because 20% of its fine-tuning captions are replaced by the empty sequence. At sampling time the same x_t is run with the caption and with empty tokens, which the released code does as one forward pass on a batch of two. The cost is two network evaluations per step instead of one.

?

What guidance scale should I use with GLIDE?
The released notebooks use s = 3.0 for text-to-image and 5.0 for inpainting. In the paper's human evaluation, the 3.5B model was rated best at s = 3.0 and the 300M model slightly better at 4.0 than at 3.0. s = 1 turns guidance off. Larger s trades diversity for caption match: in the paper's sweeps, precision and Inception Score rise while recall falls.

?

Why does GLIDE generate at 64x64 and upsample, instead of generating 256x256 directly?
A U-Net at 256x256 does 16 times as much work per layer as one at 64x64, and the base model runs 150 to 250 steps with two evaluations each. GLIDE spends that budget at 64x64, where the 3.5B model decides content and layout, and gives the 1.5B upsampler only 27 steps to add detail. The upsampler is also text-conditioned, so it can use the caption when it adds detail.

?

Can I run the 3.5B model?
No. OpenAI released GLIDE (filtered), a roughly 300M-parameter model trained on a dataset with people, violent objects and some hate symbols removed, together with its upsampler, inpainting variants and a noised ViT-B CLIP model. The 3.5B model and the noised ViT-L CLIP used in the paper's main results were not released.

?

Is the inpainting model told which pixels to keep, or does it have to guess?
It is told. The inpainting fine-tune adds four input channels: the image with the erased region zeroed, and a one-channel mask. The model therefore sees the clean surrounding pixels at every step, instead of only a noised copy of them as in the older replacement method.

Footnotes & further reading

  1. The paper: Nichol et al., GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models (ICML 2022). Code and the filtered weights: openai/glide-text2im.
  2. The diffusion model: Ho et al. (2020), in the DDPM explainer; the learned variance and cosine schedule: Nichol & Dhariwal, Improved Denoising Diffusion Probabilistic Models (2021).
  3. The U-Net and classifier guidance: Dhariwal & Nichol (2021), in the Diffusion Models Beat GANs explainer.
  4. Classifier-free guidance: Ho & Salimans (2021/2022), in the classifier-free guidance explainer.
  5. CLIP: Radford et al. (2021), in the CLIP explainer.
  6. DALL-E: Ramesh et al., Zero-Shot Text-to-Image Generation (2021); its successor, which uses a GLIDE-style decoder, in the DALL-E 2 explainer.
  7. Inpainting by replacement: Sohl-Dickstein et al. (2015) and Song et al. (2021); SDEdit: Meng et al., SDEdit (2021); trained inpainting: Saharia et al., Palette (2021).
  8. Later text-to-image diffusion: Saharia et al. (2022), in the Imagen explainer; Rombach et al. (2022), in the Latent Diffusion explainer.