High-Resolution Image Synthesis with Latent Diffusion Models

Compress images with an autoencoder, run the diffusion model on the small code, decode once at the end.

A frozen autoencoder turns a 256 × 256 color image into a 32 × 32 × 4 grid of numbers that decodes back almost unchanged. The diffusion model trains and samples on that grid, 48 times fewer numbers than the pixels, and takes text or class labels through cross-attention.

The 2021 paper behind Stable Diffusion. Most of its models train on a single A100, match or beat pixel-space diffusion models that took several times the compute, and its cross-attention conditioning is the mechanism Stable Diffusion kept.

What pixel-space diffusion costs

A diffusion model generates an image by starting from Gaussian noise and removing a little of it at a time. A single network, a U-Net called $\epsilon_\theta$, is trained to look at a noisy image and predict the noise in it; sampling calls that network once per step, for tens to hundreds of steps. The mechanics are in the DDPM explainer. By 2021 these models produced the best images on class-conditional ImageNet, with none of the training instability of GANs.

They were also expensive, because every call of the network runs on the full image. A 256 × 256 RGB image is 196,608 numbers. Training runs the network on one noised copy of each image per step, over millions of steps; sampling runs it 25 to 1000 times per image. The paper quotes the numbers for ADM, the strongest pixel-space model of the time (the Diffusion Models Beat GANs paper): 150 to 1000 V100-days to train, and about five days on one A100 to draw the 50,000 samples used to compute an FID score.

Latent Diffusion, by Robin Rombach, Andreas Blattmann and colleagues at LMU Munich, Heidelberg University and Runway ML, removes most of that cost with two stages:

1. Train an autoencoder once. Its encoder $\mathcal{E}$ maps an image to a much smaller grid of numbers, the latent, and its decoder $\mathcal{D}$ maps the latent back to an image that looks the same.
2. Freeze it, and train an ordinary diffusion model on latents instead of pixels. To generate, run the whole denoising loop on a latent and decode once at the end.

The rest of the page covers why the compression costs so little quality, how the autoencoder is trained, how far to compress, how text gets in, and what the resulting models score.

Where an image spends its bits

The case for two stages starts from a measurement on an ordinary pixel-space diffusion model. DDPM showed that a trained diffusion model can be run as a lossy compressor: send the image to a receiver a few bits at a time, in the order the denoising steps would reveal it, and after each step measure how far the receiver's best reconstruction is from the original. The paper's Figure 2 plots that measurement for DDPM's CIFAR-10 model. The horizontal axis is the rate, bits per pixel-channel received so far. The vertical axis is the distortion, the root-mean-square error of the reconstruction on the 0 to 255 pixel scale.

The curve falls from about 88 to about 9 within the first 0.2 bits per dimension, then flattens. A CIFAR-10 image has 32 × 32 × 3 = 3,072 dimensions, so those first 0.2 bits per dimension are about 77 bytes. Reaching the final error of 0.95 takes 1.78 bits per dimension, about 684 bytes. Close to 90% of the code therefore goes to moving the error from about 9 intensity levels to under 1. DDPM calls this detail imperceptible; in its words, more than half of the lossless code length describes imperceptible distortions.

The LDM authors name the two parts of the curve. The steep part is semantic compression: the bits that decide what is in the picture, its layout and objects. The flat tail is perceptual compression: bits that only fix pixel-level detail. A pixel-space diffusion model learns both with one network, and pays full price for the second part on every one of its 196,608 inputs. Drag the rate and watch where the error stops changing:

The paper builds its method on that split: a separate, cheaper model handles the perceptual compression once, producing a representation that has dropped the imperceptible detail. The diffusion model then only has to learn the semantic part, and it runs on a smaller input.

Stage one: the autoencoder

An autoencoder is an encoder and a decoder trained together so that the decoder's output matches the encoder's input. In the paper's notation the image is $x$, with height $H$, width $W$ and three color channels, and the latent is $z$:

The encoder shrinks each side by the downsampling factor $f = H/h = W/w$, a power of two. For a 256 × 256 image, $f = 4$ gives a 64 × 64 latent and $f = 8$ a 32 × 32 one. The channel count $c$ is small: the text-to-image model uses a 32 × 32 × 4 latent, 4,096 numbers against the image's 196,608, and the ImageNet model a 64 × 64 × 3 one, 12,288 numbers. The network that runs on it does its work per spatial position, so the count that matters most for cost is $h \times w$: 1,024 positions at $f = 8$ against 65,536 in pixel space, 64 times fewer.

The latent keeps the image's two-dimensional layout. Position (i, j) of $z$ describes roughly an $f \times f$ patch around pixel ($f i$, $f j$). The paper credits this layout for letting mild compression work. Earlier two-stage generators, VQGAN and OpenAI's DALL-E, also compressed images first, but then modeled the codes with an autoregressive Transformer, which reads the grid as a one-dimensional sequence and predicts one code at a time. Transformer cost grows with sequence length, so those methods compressed hard (VQGAN at $f = 16$, DALL-E at $f = 8$) and paid in reconstruction quality. LDM's denoiser is a convolutional U-Net, which is built for 2D grids and handles a 64 × 64 input cheaply, so the autoencoder can compress gently.

The paper's Figure 1 puts numbers on the difference, reconstructing ImageNet validation images: its $f = 4$ autoencoder reaches PSNR 27.4 dB and reconstruction FID 0.58; DALL-E's $f = 8$ autoencoder 22.8 dB and 32.01; VQGAN at $f = 16$ 19.9 dB and 4.98. PSNR measures per-pixel fidelity in decibels (higher is better; 27.4 dB is an average error of about 11 intensity levels out of 255). Reconstruction FID compares the statistics of reconstructed and real images in a pretrained network's feature space (lower is better; 0 means indistinguishable by that measure). Pick $f$ and the regularizer below (KL or VQ, two ways of keeping the latent's values in check: a small penalty pulling it toward a standard Gaussian, or snapping each latent vector to a learned codebook; the next section describes both); every number in the header is a row of the paper's Table 8:

Reconstruction quality falls steadily with $f$: VQ PSNR goes 30.85, 27.43, 23.07, 20.83 and 17.45 dB for $f$ = 2 to 32, and reconstruction FID climbs from 0.16 to 31.83. The drop tracks how many numbers the latent keeps. The KL model at $f = 32$ with 64 channels holds 8 × 8 × 64 = 4,096 values, the same count as the $f = 8$ models, and reconstructs at FID 2.04, far better than the 16-channel $f = 32$ models (7.3 KL, 31.83 VQ). The spatial grid is still 8 × 8, though, and the U-Net's cost and its convolutional structure depend on that grid. The paper's conditional diffusion models all use 3 or 4 channels at $f$ = 4 or 8.

Training the autoencoder

An autoencoder trained only to minimize squared pixel error produces blurry reconstructions. When several sharp textures are consistent with the latent, the output that minimizes expected squared error is their average, and the average of many textures is smooth. LDM uses the first-stage recipe of VQGAN instead, which adds two terms that penalize blur.

The first is a perceptual loss, LPIPS: pass both the original and the reconstruction through a pretrained image network and compare their intermediate features instead of their pixels. Two images with the same content and slightly shifted texture score as similar, which matches human judgment much better than pixel distance. The second is a patch-based adversarial loss: a small discriminator network, the PatchGAN from the pix2pix paper, looks at local patches and outputs, for each one, how likely it is to come from a real image. The autoencoder is trained to make its reconstructions score as real. A blurry patch is easy to tell from a real one, so this term pushes the decoder toward sharp, realistic texture. The paper describes the effect as keeping reconstructions on the image manifold (the set of images that look like real photographs).

The full objective, from the paper's appendix, is a min-max game between the autoencoder and the discriminator $D_\psi$:

$L_{rec}$ is the reconstruction loss (in the code, an L1 pixel term plus LPIPS). The discriminator, maximizing, raises $\log D_\psi(x)$ on real images and lowers its score on reconstructions; the autoencoder, minimizing, gets the reconstruction scored as real, which is what the $-L_{adv}$ term rewards. $L_{reg}$ is a regularizer on the latent.

The regularizer is there because without it nothing fixes the latent's scale: the encoder could output values of any size, spread arbitrarily, and the paper wants to avoid "arbitrarily high-variance latent spaces" before training a second model on them. The paper tries two options:

• KL-reg. makes the encoder output a mean and a variance for every latent value, as in a variational autoencoder, and adds the VAE's KL-divergence penalty pulling that Gaussian toward a standard normal, with a weight of about $10^{-6}$. A VAE normally weights this term around 1, which trades reconstruction quality for a latent that looks Gaussian. At $10^{-6}$ the penalty barely affects reconstruction and only discourages extreme values. During diffusion training the code draws $z$ from that Gaussian rather than taking its mean.
• VQ-reg. uses vector quantization, from VQ-VAE: each latent vector is replaced by its nearest entry in a learned codebook of $|\mathcal{Z}|$ vectors (8,192 or 16,384 in the models used here). LDM takes $z$ before the quantization step and treats quantization as the first layer of the decoder. The paper calls this a VQGAN with the quantization layer absorbed by the decoder. The diffusion model sees continuous latents; they get snapped to the codebook only on the way out.

VQ autoencoders reconstruct slightly worse than KL ones at the same $f$ (Table 8), but the paper reports that LDMs trained on VQ latents sometimes produce better samples. It uses VQ for the ImageNet, super-resolution and inpainting models and KL for the text-to-image model.

Setting the latent’s scale

With a KL weight of $10^{-6}$, nothing forces the latent's standard deviation to 1, and the diffusion model's noise schedule assumes data with a standard deviation near 1. The forward process mixes signal and noise as

where $\bar\alpha_t$ falls from nearly 1 at $t = 1$ to nearly 0 at $t = 1000$, and $\sigma_z$ is the standard deviation of the latent. The signal-to-noise ratio (SNR) at each step scales with $\sigma_z^2$. If the latents are five times larger than the schedule assumes, every step carries 25 times more signal relative to noise than designed, and the last step, which sampling starts from, is not close to pure noise.

The KL latents come out much larger than that, so the paper rescales them before diffusion: it estimates one standard deviation $\hat\sigma$ over all values of the first training batch and divides $z$ by it. The text-to-image config in the official code stores the result as scale_factor: 0.18215, which corresponds to a raw standard deviation of about 1/0.18215 = 5.49 for its $f = 8$ KL latents. With the paper's schedule (1,000 steps, $\beta$ from 0.00085 to 0.012), a unit-scale latent drops below SNR 1 at step 355 and keeps a signal of standard deviation 0.068 at step 1000. Unscaled, the SNR stays above 1 until step 817 and 0.375 of signal survives the last step. Slide the latent's standard deviation and watch the crossing point and the thumbnails:

The VQ latents already have a variance close to 1 and are not rescaled. The paper reports that the SNR significantly affects results when a model trained on 256 × 256 crops is run convolutionally on larger images (its Figure 15 compares KL latents with and without the rescale, and VQ latents, for semantic-map-to-image synthesis). At sampling time the code divides the final latent by the same factor before decoding.

Stage two: diffusion on the latent

With the autoencoder trained and frozen, the second stage is a standard diffusion model. The pixel-space training loss from DDPM samples an image $x$, a step $t$ uniformly from 1 to $T$, and noise $\epsilon$; builds the noised image $x_t$; and regresses the network's output on the noise:

Latent Diffusion trains the same loss on encoded images:

The only change is $x_t \to z_t$. Because the noising process has no learned parameters, $z_t$ is computed on the fly from $\mathcal{E}(x)$ during training. The network $\epsilon_\theta$ is still a time-conditional U-Net built mostly from 2D convolutions, now with a 4-channel 32 × 32 input and output instead of a 3-channel 256 × 256 one. Here is one training step of the text-to-image model with the shapes written out (the conditioning input c is covered two sections down):

# One training step of the text-to-image LDM (Eq. 3). Shapes for 256x256 images.
x    = images                        # [B, 3, 256, 256]
with no_grad():
    z = 0.18215 * E(x)               # [B, 4, 32, 32]  frozen encoder, then rescale
c    = tau(tokenize(captions))       # [B, 77, 1280]   text transformer, trained
t    = randint(1, 1000, size=B)      # one diffusion step per example
eps  = randn_like(z)                 # the noise the network must predict
z_t  = sqrt(abar[t]) * z + sqrt(1 - abar[t]) * eps
loss = mse(eps, eps_theta(z_t, t, c))
loss.backward()                      # gradients reach eps_theta and tau, not E or D

Sampling reverses this: start from Gaussian noise of the latent's shape, run the U-Net for every denoising step, and call the decoder once. The paper samples with DDIM, a deterministic sampler that skips most of the 1,000 training steps (see the DDIM explainer); its tables use between 50 and 500 steps.

# Sampling: the whole loop runs on a [4, 32, 32] tensor; D runs once at the end.
c  = tau(tokenize(prompt))           # [1, 77, 1280]
uc = tau(tokenize(""))               # empty prompt = "no condition"
z  = randn(1, 4, 32, 32)
for t in ddim_steps:                 # e.g. 250 steps (tables), 200 (Figure 5)
    e_u = eps_theta(z, t, uc)
    e_c = eps_theta(z, t, c)
    e   = e_u + s * (e_c - e_u)      # classifier-free guidance, s = 1.5 or 10
    z   = ddim_update(z, e, t)
x = D(z / 0.18215)                   # [1, 3, 256, 256], one decoder pass

Figure 4 shows the whole system. The encoder and decoder sit in pixel space and each run once per image; the denoising loop, and conditioning y, live entirely in the latent:

Two savings follow. Training and sampling both run the expensive network on 16 to 64 times fewer spatial positions (at $f$ = 4 and 8), and the autoencoder is reusable: it is trained once, on OpenImages, and then serves several diffusion models. The super-resolution model, for example, reuses the same $f = 4$ VQ autoencoder as the ImageNet experiments.

How much to compress

The downsampling factor sets both savings and losses. At $f = 1$ there is no autoencoder, and LDM-1 is pixel-space diffusion. As $f$ grows, each step gets cheaper, but the autoencoder discards more, and nothing the diffusion model learns can put back what the decoder cannot reconstruct.

The paper measures the trade on class-conditional ImageNet. It trains six LDMs with $f$ = 1, 2, 4, 8, 16 and 32, each with the same number of parameters, for the same 2 million steps, each on a single A100, and tracks FID (the distance between generated and real image statistics, lower is better, here on 5,000 samples) as training goes. Two patterns show up. Small factors, $f$ = 1 and 2, improve slowly, because the diffusion model is still doing the perceptual compression itself. Large factors stall: $f = 32$ is at FID about 95 after 400,000 steps and still near 87 after 2 million, since its autoencoder has already thrown away detail the samples need. After 2 million steps the FID gap between $f = 1$ and $f = 8$ is 38. Play the training run or scrub it; the right panel is a vertical cut through the curves at the chosen step:

The paper's own reading is that $f$ = 4 to 16 "strike a good balance between efficiency and perceptually faithful results", and, after also comparing sampling speed against FID for 10 to 200 DDIM steps (its Figure 7), that "LDM-4 and -8 offer the best conditions for achieving high-quality synthesis results". A larger factor always samples faster: with 10 DDIM steps, $f = 8$ produces about 23 samples per second against about 3 for pixel space. For complex data such as ImageNet the paper recommends the gentler $f = 4$; its text-to-image model uses $f = 8$.

Conditioning through cross-attention

A diffusion model can model a conditional distribution $p(z \mid y)$ by giving the denoiser the condition as an extra input, $\epsilon_\theta(z_t, t, y)$. For a class label that input can be a learned vector. A sentence does not fit the U-Net's grid: it is a sequence of tokens with no spatial position, of varying length. Adding one summary vector of the prompt to every grid position would lose which words matter where. The paper feeds it in with cross-attention, the attention layer a Transformer decoder uses to read its encoder's output, which lets every grid position pick out the tokens relevant to it.

First a domain-specific encoder $\tau_\theta$ turns the condition into a sequence of $M$ vectors, $\tau_\theta(y) \in \mathbb{R}^{M \times d_\tau}$. For text, $\tau_\theta$ is a Transformer over the tokenized prompt; for ImageNet classes it is a learned embedding. Then, at several resolutions inside the U-Net, the feature map is flattened into $N$ rows, $\varphi_i(z_t) \in \mathbb{R}^{N \times d_\epsilon^i}$, and attends to the condition:

The queries come from the image features and the keys and values from the condition. Row $n$ of $QK^\top$ scores grid position $n$ against every token; the softmax turns each row into weights that sum to 1; multiplying by $V$ gives each position a weighted mix of token values, which is added back to that position's features. In the text-to-image model the prompt is padded to 77 BERT tokens and encoded to width 1280, so $\tau_\theta(y)$ is 77 × 1280. At the U-Net's first level the feature map is 32 × 32 with 320 channels, so $\varphi$ is 1,024 × 320 and the attention matrix is 1,024 × 77: one distribution over the prompt for each latent position, computed in 8 heads. The same layers sit at the 16 × 16 and 8 × 8 levels. In the code each of these blocks runs self-attention over the image positions first, then the cross-attention, then a feed-forward layer. Hover over the grid to pick a position and see its attention over the prompt:

The condition encoder and the denoiser train together, end to end, with the latent loss extended by $y$:

In the text-to-image model, $\tau_\theta$ is 32 Transformer layers trained from scratch through this loss; no pretrained language model is involved beyond the BERT tokenizer. When the condition is itself an image-shaped grid (a low-resolution image for super-resolution, a masked image for inpainting, a semantic map), the paper skips attention and concatenates a resized copy of the condition to $z_t$ as extra input channels, since the positions already line up.

Classifier-free guidance

The guided benchmark numbers in Tables 2 and 3 use classifier-free guidance, from Ho and Salimans. (The appendix also reports LDM-8-G models steered by a separately trained classifier at classifier scale 10, and super-resolution samples guided by the low-resolution input.) The model is trained to make two predictions: with the condition, and with a null condition that stands for "no prompt" (the recipe drops the condition for a random fraction of training examples). The released sampling script uses the empty prompt as the null condition. At each sampling step both predictions are computed at the same $z_t$ and combined:

The difference in parentheses is the part of the prediction that the prompt changes. At $s = 0$ the prompt is ignored; at $s = 1$ the output is the plain conditional prediction; above 1 the step moves further in the prompt's direction than the conditional model alone would. In Ho and Salimans's notation the same formula has $s = 1 + w$.

The noise prediction is, up to a factor, the score of the noised latents, meaning the gradient of their log-density: $\epsilon = -\sqrt{1-\bar\alpha_t}\,\nabla_{z_t} \log p(z_t)$, with the same $\sqrt{1-\bar\alpha_t}$ as in the forward process above. A weighted sum of scores, $s\,\nabla \log p(z \mid y) + (1-s)\,\nabla \log p(z)$, is the gradient of $s \log p(z \mid y) + (1-s) \log p(z)$, so the guided prediction is the score of $p(z \mid y)^s\,p(z)^{1-s}$. By Bayes' rule $p(z \mid y)/p(z) = p(y \mid z)/p(y)$, so that density equals $p(z \mid y)\,\big(p(y \mid z)/p(y)\big)^{s-1}$: for $s > 1$ it is the conditional distribution reweighted toward points where the condition is more likely than it is on average. The figure shows this on a two-class toy, with the guidance arithmetic on the left:

The paper's ImageNet numbers show the same trade. The unguided LDM-4 has FID 10.56, precision 0.71 and recall 0.62 (precision measures how many samples look like real images, recall how much of the real distribution the samples cover). At $s = 1.25$: FID 3.95, precision 0.81, recall 0.55. At $s = 1.5$: FID 3.60, precision 0.87, recall 0.48. The benchmark tables use $s = 1.5$; the text-to-image samples in the paper's Figure 5 use $s = 10.0$, which gives vivid, on-prompt images at a much larger loss of variety. Guidance also costs a second network call per step, which lowers the ImageNet model's sampling rate from 0.7 to 0.4 samples per second (Table 18).

Results and compute

Class-conditional ImageNet 256 × 256 (Table 3). LDM-4 scores FID 10.56 with 400M parameters, against 10.94 for ADM with 554M. With classifier-free guidance at $s = 1.5$, LDM-4-G scores 3.60; ADM with classifier guidance (ADM-G, which needs a separately trained noisy-image classifier and steers by its gradient) scores 4.59 with 608M parameters. Compute differs more than parameter count. Table 18 of the paper converts its single-A100 training to 271 V100-days, against 916 for ADM (962 for ADM-G, counting its classifier), and measures sampling at 0.7 samples per second against ADM's 0.12, both with 250 steps. 271 V100-days at the paper's 2.2× A100 conversion is about 123 days on one A100.

Unconditional synthesis (Table 1). On CelebA-HQ faces at 256 × 256, LDM-4 sets a new state-of-the-art FID of 5.11 with 500 DDIM steps, better than previous likelihood-based models and GANs. On LSUN Churches, FFHQ and LSUN Bedrooms it trails the best GANs; on Bedrooms it scores 2.95 against ADM's 1.90 with about half the parameters.

Text-to-image on MS-COCO (Table 2). The 1.45B-parameter LDM-KL-8, trained on LAION-400M, scores FID 23.31 unguided and 12.63 with guidance at $s = 1.5$. The paper calls this on par with GLIDE (6B parameters, FID 12.24) and Make-A-Scene (4B, 11.84), both slightly better on FID with three to four times the parameters.

Inpainting (Tables 6 and 7). With parameter counts held equal, the latent model trains an epoch in 7.66 hours against 20.66 for the pixel-space one and reaches FID 15.21 against 24.74 after six epochs: at least 2.7 times faster and 1.6 times better in FID. A larger fine-tuned version sets a new state-of-the-art FID on Places inpainting. Table 15 lists the inpainting model as trained on eight V100s; every other conditional model there trains on a single A100.

Super-resolution (Table 5). Conditioning on the low-resolution image by concatenation, LDM-SR beats SR3 on FID (2.8 against 5.2, with FID features from the validation split) while SR3 has the better Inception Score, and a plain regression model has the highest PSNR of all, a metric that rewards blur.

Limits, and Stable Diffusion

The paper lists two limits. Sampling is still a sequential loop of network calls and slower than a GAN's single forward pass. And the decoder caps accuracy: even at $f = 4$, reconstructions are off by about 11 intensity levels on average, which can bottleneck tasks that need exact pixels. The authors expect their super-resolution models are already somewhat limited by it.

The paper's text-to-image model became the starting point for Stable Diffusion, released by the same group with Stability AI and Runway in 2022. Stable Diffusion keeps the design described here (an $f = 8$ autoencoder with a 4-channel latent, the 0.18215 scale factor and the same noise schedule, a U-Net with cross-attention, classifier-free guidance) and replaces the from-scratch text Transformer with a frozen, pretrained CLIP ViT-L/14 text encoder. Its first release has an 860M-parameter U-Net and a 123M-parameter text encoder, trained on 512 × 512 images from a subset of LAION-5B. Later work changed the denoiser rather than the latent: DiT replaces the U-Net with a Transformer over patches of the same kind of autoencoder latent.