Imagen: Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding

Encode the prompt with a frozen language model trained only on text, then generate the image with three diffusion models, from 64×64 pixels up to 1024×1024.

Imagen (Google Research, 2022) found that a bigger text encoder improved its images more than a bigger image model did, and introduced dynamic thresholding so that strong classifier-free guidance stops producing oversaturated images. It reached a zero-shot FID of 7.27 on COCO, and human raters preferred its samples to DALL-E 2's on a new prompt set, DrawBench.

Below: the diffusion objective and guidance in the paper's own notation, what a large guidance weight does to the predicted image and how static and dynamic thresholding respond, the cascade and its noise augmentation, the Efficient U-Net, and the ablations behind the claim that the text encoder matters most.

What Imagen is made of

Imagen turns a caption into a 1024×1024 image in four stages. A text encoder reads the caption and outputs one embedding vector per token. A diffusion model generates a 64×64 image conditioned on those embeddings. A second diffusion model upsamples it to 256×256, and a third to 1024×1024; both super-resolution models also see the text. Only the diffusion models are trained. The text encoder is the encoder half of T5-XXL, a language model pretrained on web text with no images, and its weights stay frozen.

Each diffusion model has a different job size. The base model produces 64 × 64 × 3 = 12,288 numbers and decides the layout, the objects and their colours. The 256×256 model produces 196,608 numbers, 16 times as many, but its input already fixes the composition; it adds detail. The last model writes 3,145,728 numbers, 256 times the base model's output. The parameter counts run the other way: 2B for the base model, 600M and 400M for the two upsamplers. Select each stage in Figure 1 to read what it consumes and how it was trained.

Two techniques make the cascade work. Classifier-free guidance, which strengthens the effect of the text at sampling time, is used with large weights in all three models, and the paper adds a sampling rule, dynamic thresholding, that keeps those weights from ruining the images. The super-resolution models are trained on deliberately corrupted low-resolution inputs, so they tolerate the flaws of the model before them.

The diffusion model in Imagen's notation

A diffusion model learns to remove noise. Training takes a clean image $\mathbf{x}$ with pixel values scaled to $[-1, 1]$, picks a time $t$ uniformly between 0 and 1, draws Gaussian noise $\boldsymbol{\epsilon}$ of the same shape, and mixes them:

At $t = 0$ the mix is the clean image ($\alpha_0 = 1$); at $t = 1$ it is pure noise. The constraint $\alpha_t^2 + \sigma_t^2 = 1$ (a "variance preserving" process) keeps the overall scale of $\mathbf{z}_t$ fixed. Imagen's base model uses the cosine schedule of Nichol and Dhariwal, under which $\alpha_t$ is close to $\cos(\pi t/2)$. At $t = 0.5$ that gives $\alpha \approx 0.703$ and $\sigma \approx 0.711$, equal parts image and noise. A pixel with value 0.6 and a noise draw of −1.2 becomes 0.703 × 0.6 + 0.711 × (−1.2) = −0.43.

A network $\hat{\mathbf{x}}_\theta$ sees $\mathbf{z}_t$, the time, and the conditioning $\mathbf{c}$ (the text embeddings, plus the low-resolution image for an upsampler) and predicts the clean image. The training loss is the squared error of that prediction, weighted by a function of time:

In practice the network outputs a guess of the noise, $\boldsymbol{\epsilon}_\theta$, and the image prediction is recovered by solving the mixing equation for $\mathbf{x}$:

Training on noise error under the cosine schedule corresponds to one particular choice of the weight $w_t$ in (1). This is the setup of DDPM, written in continuous time.

Sampling runs the process backwards. It starts from pure noise $\mathbf{z}_1$ and visits a decreasing sequence of times $1 = t_1 > \dots > t_T = 0$. At each one the network predicts the clean image $\hat{\mathbf{x}}_0^t$, and the sampler moves to a slightly less noisy point that is a function of that prediction. For the deterministic DDIM sampler (one of the two samplers the paper uses), the step from $t$ to $s < t$ is

The fraction is the noise implied by the prediction. The step keeps that noise, re-mixes it with the prediction at the lower noise level, and leaves the predicted image unchanged. Two things follow from (5). The prediction $\hat{\mathbf{x}}_0^t$ is fed straight into the next network input, so whatever range it has, the next $\mathbf{z}_s$ inherits. And a sampler can modify the prediction before taking the step; both thresholding rules below do so.

How classifier-free guidance works in Imagen

A diffusion model trained only on the conditional objective follows its text weakly: samples are varied but often miss parts of the prompt. Classifier-free guidance strengthens the conditioning at sampling time without any extra model. During training, Imagen replaces the text embeddings by zeros for 10% of examples, so the same network also learns an unconditional prediction $\boldsymbol{\epsilon}_\theta(\mathbf{z}_t)$. At sampling time it runs the network twice per step, with and without the text, and combines the two noise predictions:

The weight $w$ is the guidance weight. At $w = 1$ the formula returns the conditional prediction and guidance is off. Rewritten as $\boldsymbol{\epsilon}_u + w(\boldsymbol{\epsilon}_c - \boldsymbol{\epsilon}_u)$, it starts from the unconditional prediction and moves $w$ times the distance toward the conditional one, so any $w > 1$ extrapolates past the conditional prediction, away from what the model would draw with no prompt. The original classifier-free guidance paper writes the same rule with a weight one smaller, so Imagen's $w = 8$ is that paper's $w = 7$.

Guidance changes the predicted image in the same way. The map from a noise prediction to an image prediction, $(\mathbf{z}_t - \sigma_t \boldsymbol{\epsilon})/\alpha_t$, is affine, and (2) has weights that sum to 1, so the guided image prediction is the same extrapolation applied to the two image predictions:

Take one pixel of the red channel on a red barn. Suppose the conditional prediction is 0.7 (clearly red) and the unconditional one is 0.2 (the model, not told about a barn, predicts something greyer). At $w = 3$ the guided prediction is 0.2 + 3 × 0.5 = 1.7, and at $w = 7$ it is 3.7. Training images never had a pixel above 1. The paper confirms what the arithmetic suggests: increasing the weight improves how well images match their text but, past a point, makes them "highly saturated and unnatural".

Why large guidance weights need thresholding

The paper attributes the damage to a train-test mismatch. The network only ever saw inputs $\mathbf{z}_t$ built from images inside $[-1, 1]$. During guided sampling, a prediction of 3.7 enters the DDIM step (5), and the next input contains a signal component of $\alpha_s \times 3.7$ where training data could contribute at most $\alpha_s \times 1$. The network is then asked to denoise an input unlike anything it was trained on, its next prediction is off in turn, and because sampling applies the network to its own output at every step, the errors compound. The paper reports unnatural images and, sometimes, a sampler that diverges.

A sampler can clip the prediction elementwise back into $[-1, 1]$ before every step; the paper calls this clip static thresholding. It was already in the DDPM code, where the sampler clips the predicted image by default, but the paper knows of no study of its importance for guided sampling. Imagen finds it essential at large weights (without it, sampling can produce blank images) and still not enough: as $w$ grows, clipped samples come out oversaturated and less detailed.

Clipping a single prediction shows the problem. In the example above, every red-channel value above 1 becomes exactly 1. If half the pixels of the barn exceed 1 by different amounts, they all become the same pure red, and the differences between them, which carried the texture of the wood, are gone. The DDIM step then builds the next input from this flat, maximally saturated prediction, and the next prediction starts from there.

How dynamic thresholding works

Dynamic thresholding rescales the whole prediction instead of clipping each value. At every sampling step it takes the absolute values of all pixels and channels of the predicted image, finds their $p$-th percentile $s$ (the paper's side-by-side samples use $p = 99.5$), and if $s > 1$, clips the prediction to $[-s, s]$ and divides it by $s$:

Two cases follow. If the prediction is already in range ($s \le 1$), nothing changes. Otherwise all but the most extreme $100 - p$ percent of values are divided by the same number, which keeps their order and their ratios. The texture that clipping erased survives at a lower amplitude. Using a percentile instead of the maximum keeps one extreme pixel from setting $s$ for the whole image: the top 0.5% are clipped at $s$ and everything else is only scaled.

Figure 2 applies both rules to one guided prediction of a 32×32 barn, built as $\hat{\mathbf{x}}_u + w(\hat{\mathbf{x}}_c - \hat{\mathbf{x}}_u)$ from a conditional prediction inside the training range and a blurred, greyer unconditional one. At the default $w = 5$, the largest value is 3.42, 53% of all values lie outside $[-1, 1]$, and static thresholding pins every one of them at ±1: the barn turns a flat pure red and the grass loses its grain. Dynamic thresholding at $p = 99.5$ finds $s = 3.10$, clips the 0.5% of values beyond it, and divides the rest by 3.10. Drag $w$ back to 1 to see all three tiles agree, and push $p$ to 100 to divide by the exact maximum.

Dividing by $s$ also lowers the overall contrast of that step's prediction, which is visible in the dynamic tile at large $w$. In the full sampler this happens at every step, and the paper's samples look more natural under dynamic thresholding than under static (Figs. A.9 and A.10 of the paper show the same seed under both rules). The measured effect is Figure 3: for the 64×64 model, sweeping the guidance weight from 1 to 10 traces a curve of image quality (FID, lower is better) against text alignment (CLIP score, higher is better). The ablations compute FID from 10,000 samples of the 64×64 model, so their values are not comparable with the 30,000-sample numbers of the final results. Up to $w = 2$ the two curves nearly coincide. From $w = 3$ on, the dynamic curve sits to the right: it reaches CLIP scores near 0.292, where static thresholding peaks near 0.288, and its FID is lower than static's at the same weight up to $w = 6$. At the three largest weights its FID is a few points higher, at a much better CLIP score.

The code that implements both rules is short. The version in the paper's appendix has two bugs, fixed here and noted in the Provenance panel; the clipping happens before the DDIM step, so the next input is built from the thresholded prediction.

def guided_x0(z, t, text):                    # z: (B, 64, 64, 3)
    eps_c = eps_net(z, t, text)                  # conditional
    eps_u = eps_net(z, t, zeros_like(text))      # unconditional (text zeroed)
    eps = w * eps_c + (1 - w) * eps_u            # Eq. (2)
    a, s = alpha(t), sigma(t)
    return (z - s * eps) / a                     # x-prediction, can leave [-1, 1]

def static_threshold(x0):
    return jnp.clip(x0, -1.0, 1.0)

def dynamic_threshold(x0, p=99.5):
    axes = tuple(range(1, x0.ndim))              # every value of each image
    s = jnp.percentile(jnp.abs(x0), p, axis=axes)    # one s per image, shape (B,)
    s = jnp.maximum(s, 1.0)                      # paper prints jnp.max(s, 1.0)
    s = s.reshape(-1, *([1] * (x0.ndim - 1)))     # (B, 1, 1, 1) to broadcast
    return jnp.clip(x0, -s, s) / s

for t, t_next in zip(ts[:-1], ts[1:]):         # ts runs from 1 down to 0
    x0 = dynamic_threshold(guided_x0(z, t, text))
    eps_hat = (z - alpha(t) * x0) / sigma(t)     # noise implied by the clipped x0
    z = alpha(t_next) * x0 + sigma(t_next) * eps_hat   # DDIM step, Eq. (5)

Why a frozen text-only encoder

Earlier text-to-image models encoded the prompt with encoders trained on image-text pairs: GLIDE trained its text transformer from scratch alongside the diffusion model, and DALL-E 2 uses the text encoder of CLIP, trained contrastively to match captions with images. The natural expectation is that an encoder trained with images knows what is visually relevant. The paper tests the alternative: a language model trained on text alone, which has seen far more text (T5's C4 corpus is about 750 GB) than any captioned image dataset contains, and comes in much larger sizes.

The ablation trains a 300M-parameter 64×64 model with each frozen encoder: BERT base and large, T5-Small, Base, Large, XL and XXL (taking the encoder half and its contextual per-token outputs), and CLIP ViT-L/14 (taking the penultimate layer of its text encoder). Figure 4 shows four of the T5 curves. Each larger encoder moves the whole curve to the right and down: better text alignment and better image quality at every guidance weight. T5-Small tops out near a CLIP score of 0.261 with a best FID near 14; T5-XXL reaches about 0.285 with a best FID near 10.7.

The second tab holds the U-Net-size ablation. With T5-XXL fixed, the U-Net of the same 64×64 model grows from 300M to 2B parameters (each trained for 400K steps). The four curves nearly coincide. Measured by the best CLIP score, going from T5-Small to T5-XXL added about 0.024, and growing the U-Net 6.7 times, from 300M to 2B, added about 0.0015.

On COCO, T5-XXL and CLIP give curves that are on par. The paper argues that COCO captions are too simple to separate them and turns to DrawBench, its own set of 200 harder prompts (described in the results section). With 1B-parameter models trained on each encoder, human raters preferred the T5-XXL model's images for text alignment about 58% to 41% overall and in each of the 11 DrawBench categories, and for fidelity by a narrower margin, about 53% to 47% (both read from the paper's bar chart). The paper's interpretation, which the experiment does not test directly, is that a large language model encodes compositional prompts, such as one coloured object placed on another, better than an encoder trained to match whole captions to whole images.

Freezing also has a practical side. Because the encoder never changes, the embeddings for all training captions can be computed once and stored, so a 4.6B-parameter encoder adds almost no cost to training the diffusion models, and trying a larger encoder only means recomputing those embeddings.

How the text reaches the U-Net

The base model is a U-Net adapted from the improved-DDPM architecture: residual convolution blocks that halve the resolution on the way down (64 → 32 → 16 → 8) and double it on the way up, with skip connections between matching levels. The text enters in two ways. A pooled embedding, one vector per prompt computed by attention pooling over the tokens, is added to the timestep embedding, and that sum modulates every residual block. And at the 32×32, 16×16 and 8×8 levels, the attention layers attend over the full sequence of token embeddings.

Imagen implements this cross-attention by appending the projected text tokens to the keys and values of the existing self-attention layers. At the 16×16 level a feature map has 256 positions; with a 12-token prompt, each position's query is compared against 256 + 12 = 268 keys, so every pixel can attend to every other pixel and to every word in the same softmax. The paper also applies layer normalization to the text embeddings before the attention and pooling layers and reports that it helps "considerably".

# One attention layer of the 64x64 U-Net at the 16x16 level. Shapes for one image.
h = feature_map                                  # (16, 16, C)
img = h.reshape(256, C)                          # 256 image positions
txt = dense(layer_norm(text_emb))                # (L, C): L tokens projected to C
q = img @ Wq                                     # (256, C)
kv_in = concat([img, txt], axis=0)               # (256 + L, C)
k, v = kv_in @ Wk, kv_in @ Wv                    # (256 + L, C)
out = softmax(q @ k.T / sqrt(d_head)) @ v        # per head; each position sees
h = h + (out @ Wo).reshape(16, 16, C)            #   all pixels and all tokens

pooled = attention_pool(layer_norm(text_emb))    # (D,) one vector per prompt
cond = timestep_embedding(t) + dense(pooled)     # drives every residual block

The paper compared three ways of conditioning the base model: the mean of the token embeddings, an attention-pooled vector, and cross-attention over the sequence. Both pooled versions were noticeably worse in both image quality and alignment. A single vector has to summarise "a blue bear and a brown bird" in a fixed number of dimensions, while cross-attention lets the queries at the bear's pixel positions attend to the bear's tokens.

Super-resolution and noise augmentation

A cascade has a weakness at each hand-off. A super-resolution model is trained on pairs made from real photos: a real 256×256 image and the 64×64 image obtained by downsampling it. At sampling time its input is instead the base model's output, which has artifacts that real downsampled images do not: slightly wrong textures, faint patterns, odd colours. The upsampler was never trained on such inputs and can treat the artifacts as detail to sharpen.

Imagen uses noise conditioning augmentation, from Ho et al.'s cascaded diffusion models. During training, each low-resolution input is corrupted with Gaussian noise at a random level aug_level drawn uniformly from $[0, 1]$, in the same variance-preserving form as the forward process ($\alpha \mathbf{x} + \sigma \boldsymbol{\epsilon}$), and the model is told the level. At sampling time the base model's output is corrupted the same way at a chosen level, and the upsampler is told that level too. Noise that is stronger than the artifacts makes a generated low-resolution image statistically similar to a noisy real one, so the upsampler sees the kind of input it was trained on.

Figure 5 corrupts a real low-resolution image and the same image with a base-model-style flaw (a faint grid and a colour blotch) with the same noise. The plot tracks the ratio of signal to noise for the image content and for the flaw. Both ratios fall at the same rate, but the flaw starts about three times weaker than the image content, so it drops below the noise first. At aug_level 0.1 the flaw's ratio is 0.65 while the content's is 2.07: the layout is still readable through the noise and the flaw is not. At 0.2 the content is at 1.01, and beyond that the upsampler has to invent more and more of the picture itself.

The trade-off appears in the paper's measurements. At sampling time, aug_level = 0 gives the best FID for every guidance weight, while larger levels give slightly worse FID and a wider range of CLIP scores, meaning more varied upsampled images. For its best samples the paper uses levels from 0.1 to 0.3. A model trained without augmentation scores worse on both metrics. The paper also observes that the augmented model responds more to guidance, and suggests that noise weakens the low-resolution signal, so the model relies more on the text. With large guidance and aug_level = 0.3, changing only the super-resolution prompts (for example to "An oil painting of ...") restyles a fixed 64×64 image.

def train_step(x_lr, x_hr, text):               # x_lr 64x64, x_hr 256x256
    aug_level = uniform(0.0, 1.0)
    x_lr = alpha(aug_level) * x_lr + sigma(aug_level) * normal(x_lr.shape)
    t = uniform(0.0, 1.0)
    eps = normal(x_hr.shape)
    z_t = alpha(t) * x_hr + sigma(t) * eps          # forward process on x_hr
    loss = mse(eps_net(z_t, t, x_lr, aug_level, text), eps)

def upsample(x_lr, text, aug_level=0.2):         # x_lr from the base model
    x_lr = alpha(aug_level) * x_lr + sigma(aug_level) * normal(x_lr.shape)
    return sample(lambda z, t: eps_net(z, t, x_lr, aug_level, text))

The 1024×1024 model removes the self-attention layers, which cost memory quadratic in the number of positions, and keeps text cross-attention, which the paper found critical. Being convolutional otherwise, it trains on 256×256 crops of 1024×1024 images (with the matching 64×64 crops of the 256×256 input) and runs on the full image at inference.

The Efficient U-Net

Both super-resolution models use a U-Net variant the paper calls Efficient U-Net, which makes three changes to the usual design.

First, it moves parameters to the low-resolution levels. The 64→256 model has four levels with 128, 256, 512 and 1024 channels and 2, 4, 8 and 8 residual blocks, where standard U-Nets use 2 or 3 per level. With the downsample-first order described below, those levels run at 128, 64, 32 and 16 pixels on a side for a 256×256 input (our reading of the configuration in App. F.2). A weight in a convolution is applied once per pixel, so where it sits decides what it costs. A residual block with two 3×3 convolutions at $C$ channels has about $18C^2$ weights: 0.29M at 128 channels, 18.9M at 1024. At 16×16 each of those weights is used at 256 positions; at 128×128, at 16,384 positions. Per parameter, a block at the 16×16 level costs 64 times less compute, which is how the model grows to 600M parameters without a matching growth in memory and time.

Second, with this many residual blocks per level, the skip connections are scaled by $1/\sqrt{2}$, which the paper finds speeds up convergence significantly. Adding two uncorrelated signals of equal variance doubles the variance, so the factor keeps the scale of activations constant through the many additions (the paper does not give this reason; it is the usual justification).

Third, it reverses the order of resampling. A typical down block runs its convolutions and then downsamples; an Efficient U-Net down block downsamples first (a strided convolution) and runs its residual blocks at the lower resolution. Up blocks run their residual blocks first and upsample last. For two residual blocks of 128 channels on a 256×256 input, this cuts the cost of the 3×3 convolutions from 2 × 2 × 9 × 128² × 65,536 ≈ 38.7 billion multiply-adds to 9.7 billion, a factor of 4 (our arithmetic, counting only the 3×3 convolutions). The paper reports no quality loss from the reordering.

Against the earlier U-Net on the 64→256 task, the Efficient U-Net converges faster, reaches better final quality, and samples 2 to 3 times faster.

Training setup

The three models are trained separately; each sees real images, and the super-resolution models take a lower-resolution version of the same real image as their conditioning input. Each uses batch size 2048 for 2.5M steps. The 2B base model trains on 256 TPU-v4 chips with Adafactor, chosen because it matched Adam in early comparisons with a much smaller memory footprint; the 600M and 400M upsamplers train on 128 chips each with Adam, because Adafactor hurt their quality. Learning rate 1e-4 with 10,000 linear warm-up steps. All three use the 10% text dropout for classifier-free guidance. The base and 256×256 models use the cosine noise schedule; the 1024×1024 model uses a linear schedule with β from 1e-4 to 0.02 over 1000 steps. All are trained in continuous time.

The data combine about 460M image-text pairs from internal datasets with about 400M from the public LAION-400M dataset. The paper reports no overfitting and expects that longer training would help.

Results on COCO and DrawBench

The standard automatic benchmark is FID on COCO: generate images for 30,000 captions from the COCO validation set and compare the statistics of their Inception-network features with those of real validation images. Lower means the generated images are distributed more like real ones. "Zero-shot" means the model never trained on COCO.

• Imagen: 7.27, zero-shot, at guidance 1.35 (base) and 8.0 (super-resolution).
• DALL-E 2: 10.39 and GLIDE: 12.24, also zero-shot.
• Make-A-Scene: 7.55, trained on COCO. The best model trained on COCO in the table scores worse than Imagen without it.

FID does not fully agree with perceived quality, and CLIP score is poor at counting objects, so the paper adds human evaluation on 200 COCO captions. For photorealism, raters chose between an Imagen sample and the real reference image for the same caption; a model indistinguishable from real photos would be chosen 50% of the time. Imagen was chosen 39.5% ± 0.75%. With every caption that mentions people removed, the rate rises to 43.9% ± 1.01%, which the paper reads as a specific weakness at generating people. For alignment, raters answered whether a caption describes an image (yes = 100, somewhat = 50, no = 0): Imagen scored 91.4 ± 0.44 and the real COCO images 91.9 ± 0.42, a gap of half a point that the paper calls on par.

COCO captions are mostly simple scenes, so the paper also introduces DrawBench: 200 prompts in 11 categories that probe colours, counting, spatial relations, text rendered in the image, rare words, misspellings, long descriptions, conflicting combinations ("A horse riding an astronaut"), and prompts collected from DALL-E, Marcus et al. and Reddit. For each prompt, raters see 8 non-cherry-picked samples from each of two models and pick the better set, or "indifferent", once for fidelity and once for alignment; 25 raters per category. Figure 6 shows the aggregate preference against each competitor.

Against DALL-E 2, raters preferred Imagen about 62% of the time for alignment and 66% for fidelity. Broken down by category, Imagen wins alignment in 7 of the 11 categories and fidelity in all 11. The margins against VQGAN+CLIP and Latent Diffusion are larger (76% to 79% for alignment), and against GLIDE smaller (about 59% and 60%).

Limitations

The paper lists its weaknesses. Imagen is worse at people: the photorealism preference rises from 39.5% to 43.9% when people are removed from the test set. A preliminary internal assessment found social biases, including a tendency toward lighter skin tones and toward Western gender stereotypes for professions, and the training data include LAION-400M, which a later audit found to contain pornographic imagery, racist slurs and harmful stereotypes. The text encoder inherits the biases of its web-text pretraining. For these reasons the authors did not release code or a public demo with the paper.

Two parts of the method transferred widely. Dynamic thresholding ships in the Hugging Face diffusers schedulers (with the caveat that its sample_max_value parameter must be raised above 1 to have any effect), and DeepFloyd IF, a later open model, describes itself with the same recipe: a frozen T5 encoder and three cascaded pixel-space diffusion stages at 64, 256 and 1024 pixels. The Efficient U-Net and the claim about encoder scaling rest on ablations at 64×64 with a 300M or 1B model, and the paper does not test whether they hold at other scales. Latent diffusion took the other route to high resolution: one diffusion model on the compressed latent of an autoencoder instead of a cascade in pixel space.