GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models
A diffusion model draws images from captions, follows the caption more closely when each denoising step pushes its prediction past the captioned prediction, away from the caption-free one, and can be fine-tuned to repaint erased regions.
GLIDE (OpenAI, 2021) trained a 3.5-billion-parameter diffusion model on captioned images and compared two ways to push its samples toward the caption: the gradient of a CLIP model, and classifier-free guidance. Human raters preferred classifier-free guidance on both photorealism and caption match, and preferred its samples to DALL-E's 87% of the time on photorealism.
Explaining the paperGLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion ModelsBelow: the diffusion model and the two places the caption enters it, both guidance methods derived and then run side by side on a toy with exact scores, the Elo arithmetic behind the human evaluations, the inpainting fine-tune, and the step counts that make one image take 15 seconds.
What GLIDE is
GLIDE (Guided Language to Image Diffusion for Generation and Editing) produces a 256×256 image from a caption in two stages. A 3.5-billion-parameter diffusion model generates a 64×64 image conditioned on the caption. A 1.5-billion-parameter diffusion model, also conditioned on the caption, upsamples it to 256×256. Both are pixel-space diffusion models built on the U-Net from Diffusion Models Beat GANs (the ADM architecture). Each also contains a Transformer that encodes the caption and is trained together with the U-Net.
The paper asks one main question: what is the best way to make such a model follow its caption? It tests two methods that strengthen the effect of the caption at sampling time. CLIP guidance nudges each denoising step along the gradient of a CLIP model's image-caption similarity. Classifier-free guidance runs the model with and without the caption and extrapolates from the second prediction past the first. Human raters preferred classifier-free guidance, so every final GLIDE sample uses it.
Two more results come out of the same model. Fine-tuned with four extra input channels, it fills an erased region of a real image to match a caption and the surrounding pixels, which makes it a text-driven image editor. And a 300M-parameter version trained on filtered data, GLIDE (filtered), was released with code; the 3.5B model was not.
The diffusion model underneath
GLIDE uses the Gaussian diffusion model of DDPM. A clean image , with pixel values scaled to , is corrupted over steps. Each step shrinks the image slightly and adds Gaussian noise:
Composing the steps gives the noised image at any step directly. With ,
The released code uses the cosine schedule for the base model, under which . Halfway through, the image and the noise carry equal weight, and : a pixel with value 0.6 and a noise draw of −1.2 becomes 0.703 × 0.6 + 0.711 × (−1.2) = −0.43. By the image term is gone and is standard Gaussian noise.
Generation runs the chain backward. When each forward step adds little noise, the reverse step is close to a Gaussian, so the model predicts one:
Sampling starts from and draws one step at a time. The network does not output the mean directly. It is trained to predict the noise that was mixed in, with a squared error:
A noise prediction determines everything else. Solving the mixing equation for the image gives the model's current guess of the clean image, , and is the mean of the reverse step toward that guess. The noise prediction is also a rescaled score, the gradient of the log density of noised images:
is the exact noise prediction that approximates. The minus sign sets the direction used in guidance: the predicted noise points away from likely images, so moving against it moves the sample toward them.
The variance is learned rather than fixed, following Nichol and Dhariwal's improved DDPM. The network outputs six channels per pixel: three for and three values that interpolate, in log space, between the two natural choices of variance, the forward step's and the posterior variance . A learned variance lets the model sample well with far fewer steps than it was trained on, which GLIDE relies on: it trains on 1000 steps and samples with 27 to 250 of them.
How the caption reaches the U-Net
The caption is tokenized into a sequence of tokens (128 slots in the released code, padded) and encoded by a Transformer that is part of the diffusion model and trained with it. In the 3.5B model the Transformer has 24 residual blocks of width 2048, about 1.2B parameters; the image U-Net is the ADM ImageNet 64×64 architecture widened to 512 base channels, about 2.3B.
The Transformer's output reaches the U-Net in two ways. First, one vector summarises the caption. The final token embedding is projected and used where ADM puts its class embedding: it is added to the timestep embedding, and the sum sets a per-channel scale and shift in every residual block. Second, the whole sequence of token embeddings is projected separately for each attention layer of the U-Net and concatenated to that layer's attention context. In the code the projected tokens become extra keys and values placed in front of the image's own, so every pixel position attends over all text tokens and all other positions in one softmax.
The shapes, for the released 300M model (192 base channels, a 16-layer Transformer of width 512) whose configuration is public:
# Text path of the released GLIDE base model (text2im_model.py), shapes for one
# caption. Released config: text_ctx 128, xf_width 512, 16 layers, 192 channels.
tokens, mask = bpe(caption), pad to 128 with end_token # (128,), mask (128,)
h = token_emb(tokens) + pos_emb # (128, 512)
h = where(mask, h, padding_emb) # learned emb for pads
xf_out = layer_norm(transformer(h)) # (128, 512), no causal mask
pooled = Linear(512, 4 * 192)(xf_out[-1]) # (768,) last slot
emb = time_embed(t) + pooled # drives every res block
# inside every attention layer of the U-Net, at a level with C channels:
img = norm(feature_map).reshape(C, H * W) # one column per pixel
q, k, v = qkv(img).split(3) # each (C, H*W)
tk, tv = Conv1d(512, 2 * C, 1)(xf_out.T).split(2) # each (C, 128)
k = concat([tk, k], axis=-1) # (C, 128 + H*W)
v = concat([tv, v], axis=-1)
out = softmax(q.T @ k / sqrt(d_head)) @ v.T # per head: text + pixelsThe U-Net has attention at 32×32, 16×16 and 8×8. At 16×16, each of the 256 positions attends over 128 text keys plus 256 image keys, 384 in all; at 8×8 the text keys are two thirds of the 192. In the released code the pooled vector is the Transformer output at the last of the 128 slots. For any caption shorter than 128 tokens that slot holds a padding token, which still sees the whole caption because the text Transformer has no causal mask.
Why a text model needs guidance
A diffusion model trained on (image, caption) pairs models . Sampling from it faithfully reproduces the training distribution, including its looseness: web captions often describe only part of their image, so the images that go with any one caption vary widely, and so do the samples. In the paper's human evaluation the guided 3.5B model beats the unguided one by 156.9 Elo points on photorealism (Table 4, below). Guidance sharpens the conditional distribution at sampling time: it gives up some sample diversity to get images that match their caption more closely.
The template is classifier guidance from Dhariwal and Nichol. Train a classifier on noised images, and at each reverse step shift the mean of (2) along the gradient of its log probability for the target label:
The shift is the gradient times the step's own variance, so it is large early in sampling, when the step is uncertain, and small near the end. Applied to an unconditional model with , the shift approximately samples ; larger raises the classifier's probability to the power , which concentrates samples on images the classifier is most confident about. A classifier over 1,000 ImageNet labels is easy to train. Free-form captions have no fixed label set to train such a classifier over; the next two sections cover GLIDE's two replacements for one.
CLIP guidance with a noised CLIP
CLIP has an image encoder and a caption encoder , trained so that the dot product is high for matching pairs and low for mismatched ones. That dot product scores how well an image matches any caption, so it can stand in for the classifier's log probability in (5):
Two details make (6) work. The model being guided is already text-conditional, so CLIP guidance adds to the caption's effect rather than supplying all of it. And the CLIP model must be trained on noised images. The public CLIP models only ever saw clean photos, so for a half-noise is an input far outside their training data and its gradient is unreliable. Earlier CLIP-guided diffusion notebooks compensated with image augmentations and extra perceptual losses. GLIDE trains a noised CLIP instead: an image encoder that takes the noised image and the timestep, trained with CLIP's contrastive objective on images noised with the base model's schedule.
The noised CLIP used in the paper is a ViT-L at 64×64 with 4×4 patches, so each image becomes 16 × 16 = 256 patches. It was trained for 390K iterations at batch size 32K on an even mix of the CLIP and DALL-E datasets, then fine-tuned for 30K iterations on a broader set of internet images. In the released code the guiding gradient is that of the scaled cosine similarity: both embeddings are normalised to unit length and their dot product is multiplied by CLIP's learned logit scale before differentiating. Every guided step therefore costs a forward and a backward pass through the CLIP image encoder on top of the diffusion model's forward pass.
How classifier-free guidance works in GLIDE
Classifier-free guidance, from Ho and Salimans, removes the separate model. One diffusion network is trained to make both a conditional prediction and an unconditional one , and the sampler combines them:
At this returns the conditional prediction. For it starts from the unconditional prediction and moves times the distance to the conditional one, past the conditional prediction itself. Ho and Salimans write the same rule with a weight , so GLIDE's is their .
The "unconditional" prediction comes from the same network with an empty caption. After the main training run, GLIDE fine-tuned the base model with 20% of the token sequences replaced by the empty sequence (in the released code, 128 padding tokens with an all-false mask); the model keeps its text-conditional skill and also learns to generate with no caption.
The form of (7) comes from an implicit classifier. Bayes' rule gives , so the gradient of the log-classifier is the difference of two scores, and (4) turns each score into a noise prediction:
Now guide the unconditional score with this classifier at scale : . Multiplying through by turns the sum back into noise predictions and gives exactly (7), with exact predictions in place of learned ones. The derivation explains the form; it does not make classifier-free guidance a classifier method. The learned difference is not guaranteed to be the gradient of any function, and the guided sampler does not draw from exactly.
For one pixel at , take (the pixel from earlier) and say the conditional model predicts noise 0.30 there and the unconditional model 0.10. With the guided prediction is 0.10 + 3 × 0.20 = 0.70. The implied clean pixel is −0.92 for the conditional prediction, −0.72 for the unconditional one, and −1.32 for the guided one: guidance pushed the pixel past the conditional value and outside the valid range of . The released sampler clips the predicted clean image to at every step, which keeps such overshoots from compounding. (Imagen later replaced that clip with dynamic thresholding to allow much larger scales.)
The guided step in the released notebook:
# One guided step, as in the released notebook (text2im.ipynb). The batch holds
# the same x_t twice: first half with the caption, second half with empty tokens.
def model_fn(x_t, t, tokens, mask):
half = x_t[: len(x_t) // 2]
out = model(concat([half, half]), t, tokens=tokens, mask=mask) # 6 channels
eps, rest = out[:, :3], out[:, 3:] # rest: learned variance values
cond_eps, uncond_eps = split(eps, 2)
half_eps = uncond_eps + s * (cond_eps - uncond_eps) # s = 3.0 in the notebook
eps = concat([half_eps, half_eps])
return concat([eps, rest], dim=1) # variance taken unguided
# p_sample_loop then turns (eps, rest) into the mean and variance of
# p(x_{t-1} | x_t), clips the implied x_0 to [-1, 1], and draws x_{t-1}.Guidance touches only the three noise channels. The learned-variance channels pass through, so the step's variance is the conditional model's. The paper gives two reasons to prefer this method over CLIP guidance: the model guides itself with its own knowledge of images and captions rather than with a separate, possibly smaller model, and nothing has to be trained to classify captions, which have no fixed set of labels.
The two methods on a toy
Figure 1 runs both methods on a model small enough to compute exactly. The data are three Gaussian clusters, A, B and C, in two dimensions. The caption is "A", and the caption-conditional model is deliberately weak: it puts half its mass on A (with a narrower spread than the data) and a quarter on each of B and C, the way a model trained on loose captions does. The unconditional model is the plain three-cluster mixture. Every is computed in closed form, and 320 samples are drawn with 100 DDPM steps on the cosine schedule. For CLIP guidance the toy uses a linear scorer, and a unit vector pointing at A, and reports the same as its "CLIP score".
Without guidance, 54% of the samples land in A. Classifier-free guidance moves samples into A as grows and shrinks their spread inside it: 80% at , 99% at , where the spread has fallen from 0.40 to 0.14. That lost variation is the diversity cost the paper measures as falling recall. The CLIP gradient first does the same job faster, with 95% in A at and the spread untouched. Past that, the constant push along carries samples out of A's ring toward the upper left, where no data lives; at 56% of samples are off the data. The toy CLIP score rises the whole time, to 2.65 against a ceiling of about 1.7 for classifier-free guidance (the score of A's center), because the scorer rewards distance along whether or not the point looks like data.
The paper saw the same split at scale. It sampled MS-COCO captions at 64×64 and swept both scales, scoring the samples four ways. FID measures how far the statistics of generated images are from those of real images (lower is better); Inception Score rewards images that an ImageNet classifier labels confidently and with varied labels; precision is the share of samples that look like real data, and recall the share of real data the samples cover. On the FID-against-Inception-Score and precision-against-recall curves, classifier-free guidance is nearly Pareto optimal: no CLIP-guidance setting beats it on both axes at once. Plotted as CLIP score against FID, the order reverses: CLIP guidance raises CLIP score far more. The authors' hypothesis is the one the toy acts out, that CLIP guidance finds adversarial examples for the evaluation CLIP model, images that score well without matching the caption better. The human evaluations in the next section test it.
How people judged the samples
Each human comparison shows a rater two 256×256 images from different systems and asks one question: which looks more photorealistic, or which matches the caption better. The rater can declare neither better, which counts as half a win for each. A photorealism evaluation collects 1,000 comparisons; caption evaluations also collect 1,000, except the guidance-scale sweeps, which use 500. Wins are converted into Elo scores, the rating system from chess.
Under the Elo model, a system with score beats one with score with probability
A 400-point gap means 10:1 odds. Scores are fit by maximum likelihood: with the number of times system beat system , starting every at 0, minimise
The paper prints this objective with in the exponent. Read literally, that is the log probability that beats , and minimising it rewards losing: the fit returns the correct scores with their signs flipped. The published tables follow the standard form (10), since the system raters preferred has the highest score. Only differences of enter (9), so the scores are meaningful only relative to the other systems in the same table.
Table 1 pits the two methods against each other and against no guidance, each at its best scale from a preliminary sweep. Classifier-free guidance scores 82.7 on photorealism and 110.9 on caption similarity; CLIP guidance −73.2 and 29.3; unguided sampling −88.6 and −106.2. A gap of 155.9 points means raters would pick the classifier-free sample over the CLIP-guided one 71.0% of the time on photorealism; the caption gap of 81.6 points means 61.5%. CLIP guidance beats no guidance clearly on caption match (68.6%) but barely on photorealism (52.2%), which fits the hypothesis: it moved images toward the caption as CLIP sees it, without making them look more real.
Results against DALL-E and on COCO
Against DALL-E, OpenAI's earlier text-to-image model, a 12-billion-parameter autoregressive model trained with roughly the same compute, GLIDE wins every comparison in Table 3. The paper ran three settings, each at two DALL-E sampling temperatures (1.0 and 0.85):
| Setting | DALL-E temp. | GLIDE wins: photorealism | caption |
|---|---|---|---|
| no reranking | 1.0 | 91% | 83% |
| no reranking | 0.85 | 84% | 80% |
| DALL-E reranked with CLIP | 1.0 | 89% | 71% |
| DALL-E reranked with CLIP | 0.85 | 87% | 69% |
| reranked, GLIDE passed through DALL-E's VAE | 1.0 | 72% | 63% |
| reranked, GLIDE passed through DALL-E's VAE | 0.85 | 66% | 61% |
CLIP reranking means drawing many DALL-E samples and keeping the ones CLIP scores highest, which multiplies DALL-E's sampling cost; GLIDE uses no reranking. The last two rows push GLIDE's images through DALL-E's discrete VAE, which blurs them the way DALL-E's own outputs are blurred, to separate image sharpness from content. GLIDE still wins 66% to 72% of photorealism comparisons, so most of its lead survives the blurring. The abstract's headline figures, 87% and 69%, are the reranked row at temperature 0.85.
On MS-COCO, without ever training on it, GLIDE reaches a zero-shot FID of 12.24 (30,000 samples, reference images center-cropped). DALL-E's zero-shot FID is about 28 and LAFITE's 26.94. Models trained on COCO itself score better (lower): the GANs XMC-GAN (9.33) and LAFITE (8.12). Removing validation images that resemble training images, as the DALL-E paper did, shrinks the reference set by 21% and raises GLIDE's FID to 12.89, a change the authors attribute mostly to FID's bias on smaller reference sets.
Table 4 compares guidance with scale. Moving from the 300M model to the 3.5B model (about 12 times the parameters; the paper says roughly 10), both unguided, gains 107.9 Elo on photorealism and 109.3 on caption match. Adding classifier-free guidance to the 300M model gains 160.0 and 207.3. The guided 300M model beats the unguided 3.5B model, 57.4% of photorealism comparisons and 63.7% on caption match. Select Table 4 in Figure 2 to read these pairs.
How GLIDE edits images by inpainting
Inpainting fills a masked region of an existing image. Any diffusion model can do it without retraining by replacement: sample as usual, and after each step overwrite the known pixels with a fresh noised copy of the original, . The paper found that this occasionally leaves edge artifacts, and names the reason: the model never sees the context itself, only a noised version of it. Early in sampling the known pixels are almost pure noise, so the content generated in the hole is barely constrained by its surroundings, and late in sampling, when the context is clear, the hole's content is already decided.
GLIDE fine-tunes for the task instead. During fine-tuning, random regions of training images are erased, and the model receives the remaining pixels and a mask as extra inputs. The first convolution gets four new input channels, a second RGB image and a mask channel, whose weights are initialised to zero; at the start of fine-tuning the model computes exactly what the pretrained model did, and it learns from there how to use the context. The upsampler is fine-tuned the same way, and it always receives the full low-resolution image but only the unmasked part of the high-resolution one.
# Inpainting fine-tune (Sec. 4.3) with the input layout of InpaintText2ImUNet.
# mask = 1 where pixels are kept, 0 where the model must generate.
def train_step(x0, caption): # x0: (3, 64, 64) in [-1, 1]
mask = random_erase_mask() # (1, 64, 64)
t = randint(1, 1000); eps = randn_like(x0)
x_t = sqrt(abar[t]) * x0 + sqrt(1 - abar[t]) * eps
inp = concat([x_t, x0 * mask, mask]) # (7, 64, 64): 3 + 4 new channels
loss = mse(model(inp, t, caption)[:3], eps) # + the learned-variance term
# The first conv gets 4 extra input channels whose weights start at zero, so at
# step 0 the fine-tuned model computes exactly what the pretrained one did.
# Sampling (inpaint.ipynb) also overwrites the known pixels of each predicted x_0:
# x0_hat = x0_hat * (1 - mask) + image * maskFigure 3 shows why the clean context matters, on a 48-pixel "image" whose prior is a Gaussian process (a smooth random curve), so that both methods can use the exact score. The top panel is replacement with the exact unconditional model. The bottom panel is the ideal fine-tuned model, which samples the hole from the exact conditional distribution given the clean context; GLIDE's fine-tuned network is a learned approximation of it. Play the sampler to see what each model receives as context at each step.
At the default 14-pixel hole, the six replacement samples fan out across the hole (spread 0.77 at its center) while the conditional samples follow the erased signal (spread 0.04). Scrub back to step 35 to see why: the top panel's context is still mostly noise when the hole's coarse shape is being decided. In 2,000-sample runs of the same toy (a 14-pixel hole, 100 steps), the spread at the center was 0.94 for replacement, close to the prior's 1.0, against 0.07 for the conditional model, and the seams jumped 0.20 against 0.12. Raising the step count to 1000 does not close the gap. A 4-pixel hole is easy for both, and a 40-pixel hole leaves even the exact conditional model unsure about the middle, because the context says little about pixels far from it.
The released inpainting notebook adds one more constraint at sampling time: after each step it writes the known pixels into the model's predicted clean image, so the prediction always agrees with the context exactly. With the fine-tuned model, the paper shows edits that insert objects with matching shadows and reflections, edits to paintings that follow the painting's style, and a complex scene built by generating an image and then applying a series of inpainting edits. It also combines inpainting with SDEdit: the user paints a rough colour sketch into the masked region, the image is noised to an intermediate step, and the model denoises it into a realistic version of the sketch.
The fine-tuned model sometimes ignores the caption and fills the hole with whatever the surroundings suggest. In Appendix E the authors counter this by masking out the context channels entirely, which reduces the model to the replacement method: it then follows the caption more consistently, but its insertions fit the scene less smoothly.
What one image costs
The paper's figures sample the base model with 150 steps (100 for inpainting); its evaluations use 250 steps, which gives slightly better FID. With classifier-free guidance every step evaluates the 3.5B model twice, so a 250-step sample is 500 evaluations of the base model, run as 250 forward passes on a doubled batch.
The upsampler uses a hand-tuned 27-step schedule, found by sweeping FID. The 1000 training steps are split into five ranges of 200, and the sampler takes 10, 10, 3, 2 and 2 evenly spaced steps in them, from the least noisy range to the noisiest. The released schedule is
0, 22, 44, 66, 88, 111, 133, 155, 177, 199, 200, 222, 244, 266, 288, 311, 333, 355, 377, 399, 400, 500, 599, 600, 799, 800, 997
Twenty of the 27 steps are spent below step 400, where the upsampler adds fine detail, and two at 800 or above. The code swaps the last step, 999, for 997, with the comment that this reduces artifacts from the noisiest timesteps. The released notebook runs the upsampler with the deterministic DDIM sampler and scales the starting noise by 0.997. The unoptimised total is 15 seconds per image on one A100 GPU, against a single forward pass for a GAN.
Training used DALL-E's dataset and roughly DALL-E's compute. The base model trained for 2.5M iterations at batch size 2048, the upsampler for 1.6M iterations at batch size 512, both in 16-bit precision with standard loss scaling (as in mixed precision training).
The released model: GLIDE (filtered)
The authors judged the 3.5B model too capable of producing convincing disinformation to release. They instead trained a 300M-parameter model on a filtered dataset of several hundred million internet images, mostly disjoint from the CLIP and DALL-E data. Three filters removed images of people, of violent objects, and of two hate symbols. Each filter is an SVM with an RBF kernel on CLIP ViT-B/16 features, its bias tuned for less than 1% false negatives; on a held-out batch of 1,024 images, neither the people filter nor the violence filter let through a single false negative. GLIDE (filtered) was pretrained for 1.1M iterations and fine-tuned for 500K more for classifier-free guidance and inpainting; a filtered upsampler (192 base channels, text width 512) trained for 400K iterations. A noised ViT-B CLIP trained on filtered data was released alongside.
Red-teaming with adversarial prompts found no recognisable images of people. The filtered model still carries dataset biases, which classifier-free guidance amplifies: "toys for girls" yields more pink toys and stuffed animals than "toys for boys", and "a religious place" tends toward church-like buildings. A regurgitation test sampled 30,000 training prompts and inspected the generated images closest to their training images in CLIP space; none reproduced its training image faithfully.
Limitations
GLIDE fails on prompts that describe unusual objects or scenes; the paper's examples are "an illustration of a cat that has eight legs", "a bicycle that has continuous tracks instead of wheels", "a mouse hunting a lion" and "a car with triangular wheels". The 300M models, trained on either the full or the filtered data, bind attributes to the wrong objects more often than the 3.5B model and do worse on compositions such as stacked coloured cubes. Sampling is slow, at 15 seconds an image. The CLIP-guidance comparison rests on one noised CLIP model, and the adversarial-example explanation for its high CLIP scores is a hypothesis: the human ratings support it, and the paper does not look for adversarial structure in the images directly.
GLIDE's design was reused soon after. DALL-E 2's decoder is a modified GLIDE model conditioned on CLIP image embeddings, and classifier-free guidance, which GLIDE showed working on free-form text, became the default for text-to-image models including Imagen and latent diffusion.
Questions you might still have
Is GLIDE the model inside DALL-E 2?
Its architecture is. The DALL-E 2 (unCLIP) paper builds its image decoder by modifying the GLIDE architecture: it adds the CLIP image embedding to the timestep embedding and as four extra context tokens, and keeps the text path. The DALL-E 2 explainer covers that decoder. GLIDE itself conditions only on the caption and has no prior over CLIP embeddings.
Why did humans prefer classifier-free guidance when CLIP guidance scored higher on CLIP score?
The paper's hypothesis is that CLIP guidance finds adversarial examples: it changes pixels in whatever way raises the guiding CLIP model's similarity score, and some of those changes raise the evaluation CLIP model's score too without making the image match the caption better to a person. The evaluation used the public CLIP ViT-B/16, a different network from the noised ViT-L used for guidance, so the effect has to transfer between models. The human evaluation is the test of the hypothesis; the paper does not analyse the guided images for adversarial structure directly.
Does classifier-free guidance need a second model?
No. One network learns both predictions because 20% of its fine-tuning captions are replaced by the empty sequence. At sampling time the same x_t is run with the caption and with empty tokens, which the released code does as one forward pass on a batch of two. The cost is two network evaluations per step instead of one.
What guidance scale should I use with GLIDE?
The released notebooks use s = 3.0 for text-to-image and 5.0 for inpainting. In the paper's human evaluation, the 3.5B model was rated best at s = 3.0 and the 300M model slightly better at 4.0 than at 3.0. s = 1 turns guidance off. Larger s trades diversity for caption match: in the paper's sweeps, precision and Inception Score rise while recall falls.
Why does GLIDE generate at 64x64 and upsample, instead of generating 256x256 directly?
A U-Net at 256x256 does 16 times as much work per layer as one at 64x64, and the base model runs 150 to 250 steps with two evaluations each. GLIDE spends that budget at 64x64, where the 3.5B model decides content and layout, and gives the 1.5B upsampler only 27 steps to add detail. The upsampler is also text-conditioned, so it can use the caption when it adds detail.
Can I run the 3.5B model?
No. OpenAI released GLIDE (filtered), a roughly 300M-parameter model trained on a dataset with people, violent objects and some hate symbols removed, together with its upsampler, inpainting variants and a noised ViT-B CLIP model. The 3.5B model and the noised ViT-L CLIP used in the paper's main results were not released.
Is the inpainting model told which pixels to keep, or does it have to guess?
It is told. The inpainting fine-tune adds four input channels: the image with the erased region zeroed, and a one-channel mask. The model therefore sees the clean surrounding pixels at every step, instead of only a noised copy of them as in the older replacement method.
Footnotes & further reading
- The paper: Nichol et al., GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models (ICML 2022). Code and the filtered weights: openai/glide-text2im.
- The diffusion model: Ho et al. (2020), in the DDPM explainer; the learned variance and cosine schedule: Nichol & Dhariwal, Improved Denoising Diffusion Probabilistic Models (2021).
- The U-Net and classifier guidance: Dhariwal & Nichol (2021), in the Diffusion Models Beat GANs explainer.
- Classifier-free guidance: Ho & Salimans (2021/2022), in the classifier-free guidance explainer.
- CLIP: Radford et al. (2021), in the CLIP explainer.
- DALL-E: Ramesh et al., Zero-Shot Text-to-Image Generation (2021); its successor, which uses a GLIDE-style decoder, in the DALL-E 2 explainer.
- Inpainting by replacement: Sohl-Dickstein et al. (2015) and Song et al. (2021); SDEdit: Meng et al., SDEdit (2021); trained inpainting: Saharia et al., Palette (2021).
- Later text-to-image diffusion: Saharia et al. (2022), in the Imagen explainer; Rombach et al. (2022), in the Latent Diffusion explainer.
How could this explainer be improved? Found an error, or something unclear? I read every message.