DALL-E 2 (unCLIP): Hierarchical Text-Conditional Image Generation with CLIP Latents

Turn the caption into a CLIP image embedding first, then decode that embedding into pixels.

unCLIP, the model behind DALL-E 2, generates an image in two stages. A prior samples the CLIP image embedding that a picture matching the caption would have. A diffusion decoder, trained to invert CLIP's image encoder, turns that embedding into a 64×64 image, and two upsamplers enlarge it to 1024×1024.

The embedding fixes what the picture shows before any pixel exists. Guidance can then sharpen the pixels without pushing every sample toward the same picture, and any real photo can be encoded, varied, blended with another, or edited by a sentence.

Two stages instead of one

OpenAI's previous text-to-image diffusion model was GLIDE (December 2021), a 3.5-billion-parameter network that reads the caption and denoises a 64×64 image directly. unCLIP inserts a step in the middle. First it produces a CLIP image embedding, a vector of 1024 numbers that describes what an image contains. Then it draws an image from that vector. Write $x$ for an image, $y$ for its caption, and $z_i$ for the image's CLIP embedding. The paper's whole model is one line of probability:

The first equality holds because $z_i$ is a fixed function of $x$: run the image through CLIP's image encoder and you get exactly one embedding, so knowing $x$ already tells you $z_i$ and adding it to the event changes nothing. The second is the chain rule. Each factor on the right becomes a trained network. $P(z_i \mid y)$ is the prior, which maps a caption to an image embedding. $P(x \mid z_i, y)$ is the decoder, which maps an embedding (and, optionally, the caption) to pixels. Sampling runs them in order: draw $z_i$ from the prior, then draw $x$ from the decoder. If both networks were exact, the images would be exact samples from $P(x \mid y)$.

The authors call the result unCLIP because the decoder runs CLIP's image encoder backwards. The page covers, in order: what a CLIP embedding contains; the full sampling stack; the decoder; the prior; why this split lets guidance raise quality with less loss of variety than GLIDE suffers; the image edits that come with having an encoder and a decoder; what the decoder reveals about CLIP; the results; and the failures the authors report.

What a CLIP embedding holds

CLIP is a pair of encoders, one for images and one for text, trained on hundreds of millions of captioned images. Training pushes the embedding of each image toward the embedding of its own caption and away from the embeddings of the other captions in the batch. Both encoders output unit-length vectors, and the similarity between an image and a caption is their dot product (the cosine of the angle between them). After training, an image of a corgi and the sentence "a photo of a corgi" point in similar directions.

unCLIP trains its own CLIP for the job: a ViT-H/16 image encoder that reads 256×256 images (a Vision Transformer with 32 blocks of width 1280) and a 24-block text Transformer of width 1024, trained on about 650 million images. Both produce 1024-dimensional embeddings. The image embedding records what the image shows and its style: the objects, the scene, whether it is a photo or an oil painting. It keeps less of the exact layout and fine detail, which CLIP's training does not need to pick the right caption out of a batch. Which details survive is an empirical question, and the decoder gives a way to answer it (Section 4 of the paper, covered below).

Because of that loss, many different images share nearly the same embedding. A vector that pins down "a vase of red roses on a table, photograph" leaves the camera angle and the lighting for the decoder to choose.

How unCLIP generates an image

Figure 1 lays out the five networks that run when you type a caption: the frozen CLIP text encoder, then the prior, the decoder and two upsamplers, each trained separately. (The CLIP image encoder runs only in training and when editing an existing image.) Read the out-line in each box to follow the tensors: a caption becomes one 1024-number text embedding $z_t$, the prior turns that into one 1024-number image embedding $z_i$, the decoder turns $z_i$ into a 64×64×3 image, and the upsamplers enlarge it 4× and 4× again.

The word "hierarchical" in the title refers to this stack. The prior decides the content as one vector, the decoder renders a small image from it, and the upsamplers add resolution without seeing the caption. The trained parts total about 5.5 billion parameters: 1B for the prior, 3.5B for the decoder, and 700M and 300M for the upsamplers. The generative stack trains on the DALL-E dataset (about 250 million images). The authors also tried mixing in the noisier CLIP dataset and found it hurt sample quality.

The decoder: inverting the CLIP image encoder

The decoder is a diffusion model. Training takes a real image, adds Gaussian noise of a random strength, and trains a network to predict the noise that was added; sampling starts from pure noise and removes it step by step. (The DDPM explainer derives this.) Here the denoising network is GLIDE's 64×64 U-Net (a convolutional network with attention layers that maps an image to an output of the same size) with its original text encoder, about 2.3 billion parameters for the image part and 1.2 billion for the text Transformer, changed in two places so that it also conditions on $z_i$:

1. $z_i$ is passed through a learned linear layer and added to the timestep embedding, the vector that tells every residual block of the U-Net which noise level it is working at. Every block therefore sees the embedding.
2. $z_i$ is projected into four extra tokens appended to the caption's token outputs. Each attention layer of the U-Net attends over this list as well as over the image's own positions. With the 256-token caption context, that list has 260 entries.

A concrete training step: take a 64×64×3 image and its caption, compute $z_i$ with the frozen CLIP image encoder, pick a noise level, noise the image, and ask the U-Net to predict the noise given the noisy image, the noise level, $z_i$ and the caption. The loss is the squared error of that prediction, and the gradient updates the U-Net and its text encoder. CLIP never receives a gradient. The decoder also uses learned variances (the network predicts how much noise to inject at each sampling step as well as the mean), as in Improved DDPM, and samples with 250 strided steps (250 of the 1000 training noise levels, evenly spaced).

The conditioning is randomly removed during training: $z_i$ is set to zero (or a learned vector) 10% of the time, and the caption is dropped 50% of the time. If the two drops are independent, a step sees both inputs 45% of the time, only $z_i$ 45%, only the caption 5%, and neither 5%. These drops teach one network to make unconditional predictions as well as conditional ones, which is what classifier-free guidance needs at sampling time:

In (2), $\epsilon_\theta$ is the network's noise prediction, $c$ is the conditioning ($z_i$ and the caption), $\varnothing$ is the dropped version, and $s$ is the guidance scale. With $s = 1$ the formula returns the plain conditional prediction; with $s > 1$ it extrapolates past it, away from the unconditional prediction. This is GLIDE's convention, in which $s = 1$ means no guidance. Guidance makes samples match their conditioning more strongly and look more realistic, at the cost of variety, and the paper's main quantitative argument concerns how much variety unCLIP loses compared with GLIDE.

The two upsamplers are smaller diffusion models (ADMNet, the architecture from Diffusion Models Beat GANs) that condition only on the lower-resolution image. During training the conditioning image is corrupted, with a Gaussian blur for the 64→256 stage and, for the 256→1024 stage, the heavier BSR degradation (blur, resizing, noise and JPEG compression applied in random order), so that the upsamplers learn to repair the kind of defects the stage before them produces. Both are trained on crops a quarter of the target size and use only convolutions, so at test time they run on the full image. Captions did not help them, and they run unguided with DDIM, 27 and 15 steps.

The prior: from a caption to an image embedding

The decoder alone is an image autoencoder with CLIP as the encoder. To make images from text, something has to produce a plausible $z_i$ for a caption. The obvious candidate is the caption's own CLIP text embedding $z_t$: CLIP was trained to make $z_t$ and $z_i$ similar, so why not feed $z_t$ to the decoder in place of $z_i$?

Two facts argue against it. First, a caption describes many images. "A corgi in a party hat" fits corgis facing left and right, photos and paintings, close-ups and wide shots, and each of those has its own image embedding, while the caption has exactly one text embedding. Second, CLIP's text and image embeddings do not overlap. Liang et al. measured that the two kinds of vector sit in separate regions of the sphere, the "modality gap", so a matched pair has high similarity relative to mismatched pairs but the two vectors are still far apart. A decoder trained only on image embeddings has never seen an input from the text region.

The prior therefore models $P(z_i \mid y)$, the whole spread of image embeddings that match a caption. Figure 2 shows the geometry on a toy sphere in three dimensions instead of 1024. Each press of the button samples the prior; the teal points land at different places inside the caption's region of the image-embedding cloud, not on $z_t$.

The paper measures the alternatives directly on small models. A decoder trained to condition on CLIP text embeddings scored an FID of 9.16; a small unCLIP stack with a diffusion prior scored 7.99; and the unCLIP decoder fed text embeddings it was never trained on scored 16.55. FID (Fréchet Inception Distance) compares statistics of generated and real images in the feature space of an Inception classifier, and lower is better. In human comparisons between the first two, raters preferred the stack with the prior 57.0% ± 3.1% of the time for photorealism and 53.1% ± 3.1% for caption similarity, a consistent but modest gain.

The diffusion prior

The prior the paper prefers is itself a diffusion model, run on a 1024-number vector instead of an image. Noise is added to the embedding in the usual way, with $\bar\alpha_t$ falling from 1 to near 0 over $T = 1000$ steps of a cosine schedule:

The denoising network is a decoder-only Transformer (24 blocks, width 2048, causal attention mask) that reads a sequence of, in order: the encoded caption, $z_t$, an embedding of the timestep $t$, the noised embedding $z_i^{(t)}$, and one final slot whose output is the prediction. Where DDPM trains the network to predict the noise $\epsilon$, the authors found it better to predict the clean embedding and train with a plain squared error:

In (4), $q_t$ is the noising distribution of (3) and $f_\theta$ is the Transformer. Before any noise is added, the embedding is multiplied by 17.2, so that its scale matches the pixel values the diffusion noise schedules from ADM were tuned for (ImageNet images scaled to $[-1, 1]$). If the embeddings have unit length, a typical coordinate is about $1/\sqrt{1024} = 0.031$, and multiplying by 17.2 brings it to about 0.54, the same order as a pixel value. Without the rescale, the signal-to-noise ratio at every step of the schedule would be $17.2^2 \approx 296$ times lower than the one the schedule was tuned for.

A full training step, with shapes:

# One training step of the diffusion prior (all CLIP weights frozen)
x, y   = batch()                          # image, caption
z_i    = 17.2 * clip_image(x)             # [1024]  target image embedding
z_t    = 17.2 * clip_text(y)              # [1024]  caption's text embedding (scaled
                                          #         like z_i; the paper says "inputs")
t      = randint(1, 1000)                 # diffusion step, cosine schedule
eps    = randn(1024)
z_noisy = sqrt(abar[t]) * z_i + sqrt(1 - abar[t]) * eps
if rand() < 0.10:                         # drop text conditioning, for guidance
    y, z_t = EMPTY, NULL_EMB
seq = [*text_encoder(y),                  # 256 caption positions
       proj(z_t), time_emb(t),            # 1 + 1
       proj(z_noisy), query]              # 1 + 1   -> 260 x 2048
h = prior_transformer(seq, causal=True)   # 24 blocks, width 2048
z_pred = out(h[-1])                       # [1024]  read at the last position
loss = mean((z_pred - z_i) ** 2)          # eq. (3): predict z_i, not eps
loss.backward()                           # gradients reach the prior only

At sampling time the prior runs 64 steps with the Analytic-DPM sampler, which computes each step's optimal noise variance in closed form. It runs twice, and of the two embeddings it keeps the one with the larger dot product $z_i \cdot z_t$. That rule shifts the result toward embeddings that match the caption more closely. If the dot product of a single sample were Gaussian with standard deviation $\sigma$, the larger of two would average $\sigma/\sqrt{\pi} \approx 0.56\,\sigma$ above the single-sample mean. In Figure 2, switch best of two off and on and compare the two means in the readout. The text conditioning is dropped 10% of the time in training so the prior can be guided too, though in the aesthetic evaluation the authors found that guiding the prior hurt results, and there they guide only the decoder.

The autoregressive prior

The paper also builds a prior that writes $z_i$ as a sequence of discrete tokens, the way a language model writes text. Predicting 1024 continuous numbers one at a time is slow, so the embedding is first compressed with PCA (principal component analysis, which rotates the vector so its coordinates are ordered by how much they vary across the dataset). A CLIP model trained with SAM, a sharpness-aware optimizer, turned out to have most of its variance in few directions: keeping 319 of the 1024 principal components reconstructs the embeddings with under 1% average squared error. Each of the 319 coordinates is quantized into 1024 buckets, and a Transformer predicts the 319 tokens in order of decreasing variance, conditioned on the caption and $z_t$ as a prefix. That is 1024/319 = 3.2 times fewer tokens than coding every dimension.

The AR prior gets one extra prefix token: the quantized dot product $z_i \cdot z_t$, measured on the training pair. At sampling time the authors set that token to a value from the top half of the distribution, which asks the model for an embedding that matches the caption better than average (they swept percentiles 50, 70, 85 and 95, and 50 was best every time). The diffusion prior gets the same effect from the best-of-two rule. In every comparison in the paper the diffusion prior outperformed the AR prior at comparable model size and with less training compute.

Why guidance keeps unCLIP's diversity

Classifier-free guidance at scale $s$ approximately samples from a sharpened distribution:

At $s = 1$ this is the conditional distribution itself. For $s > 1$, every region where the conditional density is high relative to the unconditional one gets amplified, and the amplification is a power, so the more probable regions grow faster than the less probable ones. Suppose the images that fit a caption fall into a few groups (front view, side view, top view, close-up) with probabilities $\pi_k$, and the unconditional density is about the same at each group. Raising the conditional density to the power $s$ gives each group a share

With shares 45%, 28%, 17% and 10% and $s = 3$, the cubes are 0.091, 0.022, 0.0049 and 0.001, and the shares become 77%, 19%, 4% and 1%. At $s = 10$ the largest group holds 99%. This is the collapse the paper shows for GLIDE in its Figure 9: as the guidance scale rises, the camera angle, color and size of the vase converge.

unCLIP applies guidance to the decoder only, and the decoder's conditioning is a single $z_i$ that the prior already chose. The semantic choices (which view, which color) are fixed in that embedding, so guidance sharpens how the decoder renders them and cannot move a sample to another group; the paper describes the scene's semantic information as "frozen in the CLIP image embedding". In the toy, each embedding fixes the group exactly. Figure 3 runs both cases on the same toy distribution. Drag the guidance scale and watch the bars: GLIDE's shares move toward the front view, unCLIP's stay at the prior's shares while its clusters tighten.

At $s = 1$ both panels have 3.49 effective modes. At $s = 3$ GLIDE's drops to 1.99 and its bars read 77, 18, 4 and 1 percent, within a point of rule (6); at $s = 10$ it reaches 1.05, a single mode. unCLIP's stays between 3.48 and 3.49 at every scale; the slider starts at $s = 3$.

The paper's measurements match this picture. Figure 11 of the paper sweeps the decoder guidance scale and finds that guidance raises unCLIP's MS-COCO FID much less than GLIDE's; since FID penalizes a set of samples that lacks variety, a smaller rise means less collapse. Its aesthetic study (512 artistic captions written by GPT-3, four images each, scored by a linear model on CLIP features trained to predict human aesthetic ratings) finds that guidance improves the aesthetic score for both models, while for unCLIP the recall, the fraction of the training distribution the samples cover, does not drop.

Variations, blends and text diffs

Because unCLIP has an encoder (CLIP) and an approximate inverse (the decoder), it can work on existing images. The paper splits an image into two latents. $z_i$, from the CLIP image encoder, holds what CLIP recognizes. $x_T$, a noise image the size of the 64×64 decoder input (12,288 numbers), holds everything else the decoder needs to rebuild that exact picture. The paper calls the pair $(z_i, x_T)$ a bipartite latent.

$x_T$ comes from DDIM inversion. The DDIM sampler, with its randomness switched off, is a deterministic map from noise to image, so running its update in the other direction, from $t = 0$ up to $t = T$, maps an image to the noise that would regenerate it. Each DDIM step uses the network's prediction at the current noise level; reversing a step assumes that prediction is nearly the same at the neighboring level, which holds better the more steps are used. Inversion therefore reconstructs the image closely but not exactly.

Variations

To make a variation, keep $z_i$ and $x_T$ and decode with some randomness put back. DDIM has a parameter $\eta$ for this. One step from noise level $t$ down to $t-1$ is

where $\hat\epsilon$ is the network's noise prediction, $\hat x_0$ is the clean image it implies, and $\xi$ is fresh standard Gaussian noise. With $\eta = 0$ the last term vanishes and the step is deterministic, so decoding from the inverted $x_T$ retraces the inversion and returns, up to the inversion error, the input. With $\eta = 1$ the step injects as much noise as a DDPM step, and by the end the trajectory has forgotten where it started and ends at a fresh sample from $p(x \mid z_i)$. Values in between give images that keep everything stored in $z_i$ and vary more of what $x_T$ held as $\eta$ grows.

Figure 4 does this in one dimension, where the decoder's distribution for a fixed $z_i$ is four narrow modes (four images CLIP cannot tell apart) and the noise prediction is computed exactly. The amber path is the inversion of the input $x_0$; the teal paths decode from its endpoint with 14 different noise sequences.

At the default $x_0 = 0.56$ the $\eta = 0$ reconstruction lands within about 0.006 of the input. By $\eta \approx 0.6$ roughly a quarter of the decodes end in the input's mode, the same as fresh samples from four equal modes. Set $x_0$ between two modes and the reconstruction misses by much more: an input with low probability under the model does not invert cleanly. In the paper's Figure 3, variations of a painting keep its clock and its surrealism, and variations of a logo keep its overlapping strokes and color gradients, while the details CLIP does not record change.

Interpolations and text diffs

To blend two images, the paper interpolates between their embeddings $z_{i_1}$ and $z_{i_2}$ with spherical interpolation (slerp), which moves along the arc of the unit sphere between two vectors at constant speed:

for unit vectors $a$ and $b$ an angle $\Omega$ apart. A straight-line blend $(1-\theta)a + \theta b$ would cut through the inside of the sphere: halfway between two perpendicular unit vectors its length is 0.707, shorter than the unit-length embeddings CLIP produces. The paper decodes every interpolate with the same fixed $x_T$ drawn at random, so the endpoints are new images in the style of the two inputs rather than the inputs themselves; interpolating the two inverted latents as well would reproduce the inputs at $\theta = 0$ and 1.

Text diffs edit an image with a sentence. Take the CLIP text embedding of a caption describing the current image, $z_{t_0}$ ("a photo of a cat"), and of the target description, $z_t$ ("an anime drawing of a super saiyan cat, artstation"). Their normalized difference is a direction in CLIP space, and the edit rotates the image embedding toward it:

with $\theta$ raised linearly from 0 to a maximum the paper puts typically between 0.25 and 0.50, decoding each $z_\theta$ from the image's inverted $x_T$ with $\eta = 0$. The authors report that a dummy baseline caption such as "a photo", or none at all, also worked. Slerp stays in the plane of its two endpoints, so Figure 5 draws that plane exactly.

At the default 90° and $\theta = 0.40$, the edited embedding has rotated 36° toward $z_d$: its cosine with the original $z_i$ is 0.809 and with $z_d$ 0.588. At 0.50 it sits at 45°, equally similar to both (0.707). The paper stops there because $z_d$ is a difference of two text embeddings, not an image embedding; slide $\theta$ to 1 and the decoder would receive a vector of a kind it never trained on. The 1024-dimensional angle between a real $z_i$ and $z_d$ is not reported, so the angle slider is there to show how the rotation scales with it.

Using the decoder to look inside CLIP

A decoder for CLIP embeddings shows what an embedding contains: decode it several times and see what stays fixed. The paper uses this on typographic attacks, found by Goh et al., in which a paper label reading "iPod" stuck to an apple makes CLIP classify the photo as an iPod. On the paper's example, CLIP assigns 99.98% to "iPod" and 0.02% to "Granny Smith", yet decoding the image's embedding produces apples with high probability, and never an iPod. The decoded images indicate that the embedding still encodes the apple, and the label changes which caption wins CLIP's comparison.

The second probe uses PCA. The authors project embeddings onto their first 20, 30, 40, 80, 120, 160, 200 and 320 principal components and decode each truncation with a fixed seed. The first few dozen components carry coarse content (there is food and maybe a container); later ones add specifics (tomatoes, a bottle) and the shapes of objects. Since the AR prior predicts the components in this order, the sequence also shows what that prior commits to first.

Results against GLIDE and on MS-COCO

Human raters compared unCLIP with GLIDE in pairs, choosing which image was more photorealistic, which matched the caption better, and, for a new test, which of two 4×4 grids of samples for the same caption was more diverse. Each number below is the probability that raters picked unCLIP over GLIDE, so 50% is a tie (Table 1 of the paper, 95% intervals):

With the diffusion prior, photorealism is a statistical tie (the interval contains 50%), caption similarity favors GLIDE (the interval ends at 48.3%), and raters picked unCLIP's grids as more diverse 70.5% of the time. A natural objection is that GLIDE could buy diversity back by guiding less. The paper tested that (its Figure 10): GLIDE at guidance scale 2.0 comes close to unCLIP in photorealism and caption similarity and is still less diverse.

On zero-shot MS-COCO, where models generate images for captions from the COCO validation set without training on COCO, unCLIP set a new best FID at 256×256 (Table 2, decoder guidance scale 1.25):

Models trained on COCO itself score lower (Make-A-Scene reaches 7.55), so 10.39 is a record only among zero-shot models. Against GLIDE, whose architecture the unCLIP decoder reuses, the improvement is 12.24 to 10.39.

Limitations

The paper reports three failures, each traced to the CLIP embedding or to the 64×64 base resolution:

• Attribute binding. For "a red cube on top of a blue cube", unCLIP mixes up the colors more often than GLIDE. The paper's hypothesis is that the CLIP embedding does not bind attributes to objects, and its evidence is that decoding the CLIP embedding of a real image of two colored objects often swaps the colors, or in one example the relative sizes. The GLIDE caption pathway in the decoder does not fix this.
• Text in images. For "a sign that says deep learning", the signs carry misspelled text. The CLIP embedding may not record spelling, and the BPE tokenizer (which splits text into subword tokens) hides the letters of each word from the model, so it has to learn to render a token only from images where that token appears written out.
• Detail in complex scenes. Crowded scenes such as Times Square come out with little fine detail, which the authors attribute to starting at 64×64 and upsampling. A higher base resolution should help, at a higher training and sampling cost.

On risk, the paper notes that unCLIP's better samples raise the risk of deceptive images above GLIDE's, and points to the system card for the DALL-E 2 Preview, the first deployment of an unCLIP model, for an analysis in that deployment's context.

Sampling, variations and text diffs in pseudocode

Sampling, variations and text diffs, with the paper's step counts. The functions are named for what they do; OpenAI released no code for unCLIP.

# Sampling one 1024x1024 image for caption y (paper models, App. C)
z_t = clip_text(y)                               # frozen
cands = [diffusion_prior.sample(y, z_t, steps=64)   # Analytic-DPM sampler
         for _ in range(2)]
z_i = max(cands, key=lambda z: dot(z, z_t))      # keep the better-matching one
x64 = decoder.sample(z_i, y, steps=250,          # 3.5B GLIDE-style U-Net,
                     guidance=s)                  # classifier-free guided
x256 = up1.sample(x64, steps=27)                 # DDIM, no text, no guidance
x1024 = up2.sample(x256, steps=15)               # DDIM, no text, no guidance

# Variation of an existing image x (Sec. 3.1)
z_i = clip_image(x)
x_T = ddim_invert(decoder, x, cond=z_i)          # approximate inversion
x_var = decoder.sample(z_i, x_T=x_T, eta=0.5)    # eta = 0 reconstructs x

# Text diff (Sec. 3.3): "a photo of a cat" -> "an anime drawing of a cat"
z_d = normalize(clip_text(y_new) - clip_text(y_now))
for th in linspace(0, 0.5, 6):
    out = decoder.sample(slerp(z_i, z_d, th), x_T=x_T, eta=0)

Latent diffusion (the basis of Stable Diffusion), posted four months earlier, also runs diffusion in a learned representation, but that representation is a spatial grid from an autoencoder trained to reconstruct pixels, not a single semantic vector, so it keeps the layout that a CLIP embedding discards.