DDPM: Denoising Diffusion Probabilistic Models

Destroy an image with noise, then learn to reverse it.

A fixed process adds a little Gaussian noise to an image, 1000 times, until only static is left. A single network is trained to predict the noise in a partly noised image. Sampling starts from fresh static and uses that network 1000 times to remove noise step by step.

Ho, Jain and Abbeel took a 2015 model class that had never produced good samples, reduced its training loss to a mean-squared error on Gaussian noise, and reached an FID of 3.17 on CIFAR-10.

In 2020 the sharpest generated images came from GANs (see the GAN explainer): a generator trained against a discriminator. GAN training is a two-player game with no single loss going down, and it often collapses onto a few kinds of output. Likelihood-based models (autoregressive models, normalizing flows, and variational autoencoders) train by minimizing one well-defined loss, but their samples were visibly worse. A third family, diffusion probabilistic models, had been proposed by Sohl-Dickstein et al. in 2015, and as this paper puts it, there had been no demonstration that they could generate high-quality samples.

The method has two halves. The forward half takes a training image and adds a small amount of Gaussian noise, then a little more, for 1000 steps, until it cannot be told apart from random static. This half has no parameters. The learned half is one neural network that takes a noisy image and the step number and predicts the noise that was added. To generate, start from pure static and apply that network 1000 times, each time subtracting a little of the predicted noise and adding back a smaller amount of fresh noise.

Five ideas explain why this works and why the loss is so short: the forward noising chain; a closed form that noises an image to any step in one draw; a reverse chain trained with a variational bound; the Gaussian target inside that bound, which is available only because the clean image is known during training; and the change of variables that turns the bound into a mean-squared error on noise. After those come sampling, the code, and the results. The headline result is an FID of 3.17 on unconditional CIFAR-10. FID (Fréchet Inception Distance) compares the statistics of generated and real images in the feature space of an image classifier; lower is better, and 3.17 was better than every unconditional model the paper compares against.

The forward process adds noise on a fixed schedule

Treat an image as a vector. A CIFAR-10 image is $32\times32\times3$, so it is a vector $\mathbf{x}_0$ of $D=3072$ numbers, with pixel values in $\{0,\dots,255\}$ scaled linearly to $[-1,1]$. The forward process (the paper also calls it the diffusion process) is a Markov chain $\mathbf{x}_0 \to \mathbf{x}_1 \to \dots \to \mathbf{x}_T$ with $T=1000$ steps: each image in the chain is computed from the previous one alone. Each step is one Gaussian:

One step scales the previous image down slightly, by $\sqrt{1-\beta_t}$, then adds independent Gaussian noise with variance $\beta_t$ to every pixel. The numbers $\beta_1,\dots,\beta_T$ are the variance schedule. DDPM fixes them in advance and never learns them: $\beta_t$ rises linearly from $\beta_1=10^{-4}$ to $\beta_T=0.02$. The paper chose these values to be small relative to data in $[-1,1]$, so that each reverse step stays close to Gaussian, while leaving as little signal as possible at $\mathbf{x}_T$. It set $T=1000$ without a sweep, so that sampling uses the same number of network evaluations as earlier work. The whole chain factors into those steps:

The scale factor $\sqrt{1-\beta_t}$ keeps the variance bounded. Suppose a pixel of $\mathbf{x}_{t-1}$ has variance $v$. After one step its variance is $(1-\beta_t)\,v + \beta_t$, which equals 1 when $v=1$ and moves toward 1 from either side otherwise. Without the scale factor each step would add $\beta_t$ and nothing would remove any, so after 1000 steps the variance would have grown by $\sum_t\beta_t = 10.05$ and the endpoint would depend on the schedule's total. With it, every image ends near the same distribution, a standard Gaussian $\mathcal{N}(\mathbf{0},\mathbf{I})$, which is where sampling will start. Later work named this shrink-then-add design "variance-preserving"; the name comes from Song et al.'s SDE paper, not from DDPM.

Drag the step below and watch a point cloud shaped like the letters "ip" shrink toward the origin while noise spreads it out. The two numbers in the corner are the signal and noise weights that the next section derives.

Noising to any step in one draw

Training will need noisy images at randomly chosen steps, for example $\mathbf{x}_{700}$. Running 700 sequential steps for every training example would be slow. It is also unnecessary, because a chain of Gaussian steps composes into a single Gaussian. Define $\alpha_t := 1-\beta_t$, the per-step factor, and $\bar\alpha_t := \prod_{s=1}^{t}\alpha_s$, the running product. Then

Two steps show where this comes from. Write each step with its own standard-normal noise:

The signal coefficient is $\sqrt{\alpha_1\alpha_2}=\sqrt{\bar\alpha_2}$. The two noise terms are independent, so their variances add: $\alpha_2\beta_1+\beta_2$. Substituting $\beta=1-\alpha$ gives

and the same substitution works at every further step, so the noise variance after $t$ steps is $1-\bar\alpha_t$. In the form used in code, draw one $\boldsymbol\epsilon$ and compute

The squares of the two weights, $\bar\alpha_t$ and $1-\bar\alpha_t$, always sum to 1. Their ratio $\bar\alpha_t/(1-\bar\alpha_t)$ is the signal-to-noise ratio (SNR). With the paper's schedule:

• $t=1$: $\bar\alpha=0.9999$, noise weight 0.010. The image is untouched to the eye.
• $t=100$: $\bar\alpha=0.897$, signal weight 0.947, noise weight 0.321, SNR 8.7.
• $t\approx259$: $\bar\alpha=0.5$, both weights 0.707, SNR 1.
• $t=500$: $\bar\alpha=0.0786$, signal weight 0.280, noise weight 0.960.
• $t=1000$: $\bar\alpha\approx4.04\times10^{-5}$, signal weight 0.0064.

At $t=1000$ a pixel at +1 contributes 0.0064 to a sum whose noise has standard deviation 0.99998. The paper computes the leftover mismatch between $q(\mathbf{x}_T\mid\mathbf{x}_0)$ and $\mathcal{N}(\mathbf{0},\mathbf{I})$ as about $10^{-5}$ bits per dimension. Drag the step and read the SNR; notice that SNR 1 arrives at about a quarter of the chain, so most steps work on images that are more noise than signal.

For training, (4) turns "noise this image to step 700" into one multiply-add over 3072 numbers with $\bar\alpha_{700}$ read from a precomputed table of 1000 values.

The reverse process and the bound that trains it

Generation runs the chain the other way: start at $\mathbf{x}_T\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ and step down to $\mathbf{x}_0$. The exact reverse step $q(\mathbf{x}_{t-1}\mid\mathbf{x}_t)$ is unknown. To compute it you would have to average over every clean image that could have produced $\mathbf{x}_t$, weighted by how likely each is, and that requires the data distribution the model is trying to learn.

DDPM approximates each reverse step with a Gaussian. The justification is a result the paper takes from Sohl-Dickstein et al.: when every $\beta_t$ is small, the forward and reverse processes have the same functional form, so the true reverse step is close to Gaussian. The largest $\beta_t$ here is 0.02. The reverse chain, or reverse process, is

with $p(\mathbf{x}_T)=\mathcal{N}(\mathbf{0},\mathbf{I})$. A network computes the mean $\boldsymbol\mu_\theta$. DDPM does not learn the covariance: it sets $\boldsymbol\Sigma_\theta=\sigma_t^2\mathbf{I}$ with $\sigma_t^2$ a fixed number per step, because learning it made training unstable and samples worse; the two fixed values it uses appear in the sampling section.

Training uses the same tool as a variational autoencoder. The noisy images $\mathbf{x}_{1:T}$ play the role of latent variables, the forward process plays the role of the encoder (fixed, with no parameters), and the reverse process is the decoder. Jensen's inequality gives an upper bound on the negative log-likelihood, which training minimizes:

Minimizing $L$ is the same as maximizing the evidence lower bound (ELBO) of a VAE; only the sign differs. Both chains factor over steps, and after a rewrite with Bayes' rule (the paper's Appendix A, from Sohl-Dickstein et al.) the bound splits into one term per step. Each term is a KL divergence $D_{\mathrm{KL}}$, a measure of how far one distribution is from another that is zero only when the two match:

$L_T$ contains no parameters, since the forward process is fixed and $p(\mathbf{x}_T)$ is a fixed Gaussian; it is the $10^{-5}$ bits per dimension quoted above, a constant that training ignores. $L_0$ scores the last step as a discrete decoder that assigns probability to each of the 256 pixel values (the paper's equation (13)). The remaining 999 terms each compare the model's reverse step with $q(\mathbf{x}_{t-1}\mid\mathbf{x}_t,\mathbf{x}_0)$: the reverse step of the forward process when the clean image is also given. Conditioning on $\mathbf{x}_0$ makes the target computable, as the next section shows.

The target: the reverse step given x₀

Given both the clean image $\mathbf{x}_0$ and the noisy $\mathbf{x}_t$, the distribution of the in-between $\mathbf{x}_{t-1}$ is proportional to $q(\mathbf{x}_t\mid\mathbf{x}_{t-1})\,q(\mathbf{x}_{t-1}\mid\mathbf{x}_0)$, a product of two Gaussians in $\mathbf{x}_{t-1}$, so it is itself Gaussian. This forward-process posterior is

The mean is a weighted sum of the clean image and the noisy one. The $\mathbf{x}_t$ coefficient uses the per-step $\sqrt{\alpha_t}$, the $\mathbf{x}_0$ coefficient uses the cumulative $\sqrt{\bar\alpha_{t-1}}$. With the paper's schedule:

• $t=2$: $\tilde{\boldsymbol\mu}=0.545\,\mathbf{x}_0+0.455\,\mathbf{x}_2$. Early in the chain one step moves about halfway to the clean image.
• $t=10$: $\tilde{\boldsymbol\mu}=0.147\,\mathbf{x}_0+0.853\,\mathbf{x}_{10}$, $\tilde\beta_{10}=0.000238$ against $\beta_{10}=0.000279$.
• $t=500$: $\tilde{\boldsymbol\mu}=0.0031\,\mathbf{x}_0+0.9941\,\mathbf{x}_{500}$, $\tilde\beta_{500}=0.01003$ against $\beta_{500}=0.01004$. Late in the chain one step moves very little.

With the model's variance fixed at $\sigma_t^2$, the KL divergence between two Gaussians in (5) reduces to a scaled squared distance between their means, plus a constant $C$ that does not depend on $\theta$:

Equation (8) is a regression: the network sees $\mathbf{x}_t$ and $t$, and its output is scored against a target that also depends on $\mathbf{x}_0$, which the network never sees. At sampling time there is no $\mathbf{x}_0$. For squared-error regression, the loss at each input is minimized by the average of the targets that occur with that input. Many clean images could have produced a given $\mathbf{x}_t$. Averaging $\tilde{\boldsymbol\mu}_t(\mathbf{x}_t,\mathbf{x}_0)$ over them, weighted by $q(\mathbf{x}_0\mid\mathbf{x}_t)$, gives exactly the mean of the true reverse step, because

So the network is trained with targets it can compute only because $\mathbf{x}_0$ is known, and it converges toward a step that does not need $\mathbf{x}_0$. The true reverse step is a mixture of Gaussians, one per possible clean image, and the model step is a single Gaussian with the mixture's mean. The figure makes this exact on a dataset with two points, $x_0=-1$ and $x_0=+1$, in one dimension. It shows the two conditional Gaussians (6), each scaled by how likely its $x_0$ is given $x_t$, their sum, which is the true reverse step, and the single Gaussian a DDPM step can represent. The jump slider generalizes (6) from a one-step jump to a $k$-step jump, from $t$ to $s=t-k$, using the same formula with $\bar\alpha_s$ and $\bar\alpha_t/\bar\alpha_s$ in place of $\bar\alpha_{t-1}$ and $\alpha_t$.

The overlap readout measures how much probability the Gaussian model step shares with the true step. With $k=1$ it stays near 100% at every $t$ and $x_t$, which is the small-$\beta$ argument for a Gaussian reverse step, checked numerically. From $t=500$ at the default $x_t$, a 100-step jump still overlaps 96%, a 250-step jump 79%, a jump to $s=100$ 21%, and a jump to $s=1$ 0%. Jumps that stay at high noise remain nearly Gaussian (from $t=1000$ to $s=645$ the overlap is above 99%), because both data points are still buried in noise there. The Gaussian form fails when one step spans the range of noise levels over which the data points become distinguishable, which is why DDPM keeps all 1000 small steps.

Predicting the noise instead of the mean

The most direct choice is a network that outputs $\tilde{\boldsymbol\mu}_t$. The paper instead rewrites the target in terms of the noise. From (4), the clean image is determined by $\mathbf{x}_t$ and the noise that was added:

Substitute this into $\tilde{\boldsymbol\mu}_t$ from (7). Two terms now multiply $\mathbf{x}_t$, and their coefficients add to $\big(\beta_t+\alpha_t(1-\bar\alpha_{t-1})\big)/\big(\sqrt{\alpha_t}(1-\bar\alpha_t)\big)$. The numerator is $1-\alpha_t+\alpha_t-\bar\alpha_t=1-\bar\alpha_t$, which cancels the denominator's $1-\bar\alpha_t$ and leaves $1/\sqrt{\alpha_t}$. The target mean becomes $\tfrac{1}{\sqrt{\alpha_t}}\big(\mathbf{x}_t-\tfrac{\beta_t}{\sqrt{1-\bar\alpha_t}}\boldsymbol\epsilon\big)$, and the only unknown in it is $\boldsymbol\epsilon$. So DDPM trains a network $\boldsymbol{\epsilon}_\theta(\mathbf{x}_t,t)$ to predict the noise and assembles the mean from it:

At $t=500$ this reads $\boldsymbol\mu_\theta = 1.00506\,(\mathbf{x}_t - 0.01046\,\boldsymbol\epsilon_\theta)$: subtract about 1% of the predicted noise vector, then undo that step's shrink by multiplying by $1/\sqrt{\alpha_{500}}$. The prefactor uses the per-step $\alpha_t$; the noise coefficient uses the cumulative $\bar\alpha_t$, because $\boldsymbol\epsilon_\theta$ estimates all of the noise accumulated since $\mathbf{x}_0$, and a single step removes only its share.

Predicting $\boldsymbol\epsilon$, predicting $\mathbf{x}_0$, and predicting $\tilde{\boldsymbol\mu}_t$ carry the same information once $\mathbf{x}_t$ is known. The figure draws (4) for a single point: the clean $\mathbf{x}_0$ is scaled to $\sqrt{\bar\alpha_t}\,\mathbf{x}_0$, and the noise arrow $\sqrt{1-\bar\alpha_t}\,\boldsymbol\epsilon$ carries it to $\mathbf{x}_t$. The network's output is $\boldsymbol\epsilon$, that arrow's direction and length before scaling.

Substituting (11) into (8) gives a squared error between the true and predicted noise, with a weight that depends on $t$:

The paper's main training objective drops that weight:

with $t$ uniform on $\{1,\dots,1000\}$. A training step samples an image, a step and a noise vector, builds $\mathbf{x}_t$ with one use of (4), and regresses the network's output onto the noise. Nothing is sampled from the model during training, and no chain is unrolled. The regression target $\boldsymbol\epsilon$ has unit variance at every $t$, so the network's output has the same scale at every noise level. A $\tilde{\boldsymbol\mu}_t$ target is nearly a copy of the input (at $t=500$ it is $0.9941\,\mathbf{x}_t+0.0031\,\mathbf{x}_0$), so the informative part of the output would be a small correction on top of that copy. The authors also tried predicting $\mathbf{x}_0$ and found it gave worse samples early in their experiments.

The ε-parameterization also connects DDPM to score-based models. The score of a density is the gradient of its log, which points toward higher probability. From (4), $\nabla_{\mathbf{x}_t}\log q(\mathbf{x}_t\mid\mathbf{x}_0) = -\boldsymbol\epsilon/\sqrt{1-\bar\alpha_t}$, so $\boldsymbol\epsilon$ points away from the clean image and $-\boldsymbol\epsilon_\theta/\sqrt{1-\bar\alpha_t}$ estimates the score of the noisy data. That the least-squares denoiser recovers the score of the noisy density is Tweedie's formula (Robbins, 1956); DDPM does not cite it. The paper's own statement is that (12) resembles denoising score matching over multiple noise scales indexed by $t$ (training a network on noised data to output the score of the noised density), the objective of Song and Ermon's NCSN, and that its sampler resembles Langevin dynamics (repeatedly stepping along the score and adding noise) with $\boldsymbol\epsilon_\theta$ as a learned gradient of the data density.

What dropping the weights changes

With $\sigma_t^2=\beta_t$, the weight in (12) simplifies to $w_t = \beta_t/\big(2\alpha_t(1-\bar\alpha_t)\big)$. It is largest at the start of the chain, where $1-\bar\alpha_t$ is tiny: $w_2=0.273$, $w_{100}=0.0101$, a minimum of 0.00497 near $t=349$, and $w_{1000}=0.0102$. Step 2 is weighted 27 times more than step 1000. The true bound therefore spends much of its training signal on steps where the image has almost no noise, and $L_{\text{simple}}$ spreads it evenly.

The figure plots the cumulative share of the weight held by steps 2 through a cutoff $c$ (step 1 is the decoder term $L_0$ and is left out). At $c=100$, 10% of the steps, the bound puts 34.1% of its weight there with $\sigma_t^2=\beta_t$ and 39.2% with $\sigma_t^2=\tilde\beta_t$; $L_{\text{simple}}$ puts 9.9% there, the share of steps.

The paper's explanation: the small-$t$ terms train the network to remove very small amounts of noise, so down-weighting them lets the network focus on the more difficult denoising tasks at larger $t$. The paper calls $L_{\text{simple}}$ a weighted variational bound; it is no longer the tightest bound on likelihood. The ablation on CIFAR-10 (Table 2, Inception Score / FID) measures both effects. Fixed isotropic $\boldsymbol\Sigma$ means $\sigma_t^2\mathbf{I}$ with a constant $\sigma_t^2$:

• ε-prediction with $L_{\text{simple}}$: 9.46 / 3.17.
• ε-prediction on the true bound, fixed isotropic $\boldsymbol\Sigma$: 7.67 / 13.51.
• $\tilde{\boldsymbol\mu}$-prediction on the true bound, fixed isotropic $\boldsymbol\Sigma$: 8.06 / 13.22; with a learned diagonal $\boldsymbol\Sigma$: 7.28 / 23.69.
• $\tilde{\boldsymbol\mu}$-prediction with an unweighted squared error, and ε-prediction with a learned $\boldsymbol\Sigma$: unstable to train, with poor samples and out-of-range scores.

Predicting $\tilde{\boldsymbol\mu}$ works only on the true bound; predicting ε works on the true bound and works much better on $L_{\text{simple}}$. The likelihood moves the other way. The $L_{\text{simple}}$ model's negative log-likelihood is at most 3.75 bits per dimension on the test set; the model trained on the true bound reaches at most 3.70. FID drops from 13.51 to 3.17 for a 0.05 bits/dim loss in likelihood.

Sampling with Algorithm 2

To generate an image, draw $\mathbf{x}_T\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ and for $t=T,\dots,1$ compute the mean (11) from the network's noise prediction and add fresh noise:

The coefficient $1-\alpha_t$ is $\beta_t$, so the first term is (11). The last step adds no noise and returns the mean. For $\sigma_t^2$ the paper tried two values and found similar results: $\sigma_t^2=\beta_t$, which is optimal if the data were a standard Gaussian, and $\sigma_t^2=\tilde\beta_t$, which is optimal if the data were a single fixed point. The two are nearly equal for most of the chain (0.01004 and 0.01003 at $t=500$).

At any step the sampler also has an estimate of the final image. Rearranging (4) with the predicted noise gives

which the paper tracks while sampling CIFAR-10 images: large-scale features appear first and details appear last. Early in the chain $\mathbf{x}_t$ is mostly noise, so the best estimate is an average over many possible images; each step narrows that set.

The figure runs Algorithm 2 on six points in 2D, arranged as two groups of three. In place of a trained network it uses the exact optimal noise predictor for this dataset, so any failure in the figure comes from the sampler, not from training error. The default view is the end of a 1000-step run; press Play to watch it from $\mathbf{x}_T$, and turn on show x̂₀ to see each particle's current estimate (15). The estimates first sit at the centre of the data, then split into the left and right groups, then separate into single points.

The step-count slider compares models with different $T$. Each uses the paper's $\bar\alpha$ curve sampled at $T$ evenly spaced points, which amounts to a separately trained model per $T$; an exact predictor makes that free in the toy. With $T=1000$ every particle lands on a data point and all six are reached. With $T=10$ about 80% land on one. With $T=1$ every particle lands at the same place near the centre: one step from pure noise returns $\hat{\mathbf{x}}_0$ at $t=1000$, where $\mathbf{x}_T$ carries almost no information, so the estimate is the average of the data. Large jumps fail for the reason Figure 3 shows: the true reverse step becomes multimodal, and a Gaussian with the right mean puts its mass between the modes.

Set $\sigma_t=0$ and keep all 1000 steps, and every particle ends on one of the two inner points; the four outer points get none. Each mean step pulls particles toward the current $\hat{\mathbf{x}}_0$ and removes some of their spread, and $\sigma_t\mathbf{z}$ puts back the spread that the true reverse step has. Without it the cloud contracts toward the centre while $\hat{\mathbf{x}}_0$ is still the average of all six points, and by the time the points are distinguishable every particle is closest to an inner one. The DDIM sampler reaches a deterministic update by changing the coefficients as well as setting the noise to zero, which is a different update from this one.

Each step is one network call and the steps are sequential, so an image costs 1000 forward passes. The paper reports 17 seconds for a batch of 256 CIFAR-10 images and 300 seconds for a batch of 128 images at $256\times256$, on a TPU v3-8 (similar to 8 V100 GPUs). A GAN produces a batch in one forward pass.

Training and sampling in code

The network $\boldsymbol\epsilon_\theta$ is a U-Net similar to an unmasked PixelCNN++, built from Wide ResNet blocks with group normalization and self-attention at the $16\times16$ resolution. It takes a $3\times32\times32$ noisy image and returns a tensor of the same shape, one noise estimate per pixel value. The same weights serve every step; the step $t$ enters as the sinusoidal position embedding from the Transformer, added inside each residual block. The CIFAR-10 model has 35.7 million parameters, and the LSUN and CelebA-HQ models have 114 million.

Take one CIFAR-10 image, 3072 values in $[-1,1]$, and suppose the sampled step is $t=400$. The table gives $\bar\alpha_{400}\approx0.195$, so the signal weight is 0.44 and the noise weight is 0.90: $\mathbf{x}_{400}=0.44\,\mathbf{x}_0+0.90\,\boldsymbol\epsilon$, already mostly noise. The U-Net receives $(\mathbf{x}_{400}, 400)$ and returns 3072 numbers; the loss is their mean squared difference from $\boldsymbol\epsilon$. The gradient flows into the U-Net's weights only; $\mathbf{x}_{400}$ is data, built without any parameters. In a batch, every image gets its own $t$ and $\boldsymbol\epsilon$:

# Algorithm 1: one training step (batch of B images, D = 3072 values each)
x0   = next_batch()                    # (B, 3, 32, 32), scaled to [-1, 1]
t    = randint(1, T + 1, size=B)       # one uniform step per image
eps  = randn_like(x0)                  # the target
a    = abar[t].view(B, 1, 1, 1)        # abar precomputed for t = 1..1000
xt   = sqrt(a) * x0 + sqrt(1 - a) * eps          # eq (4); no gradient here
loss = ((eps - eps_theta(xt, t)) ** 2).mean()   # L_simple, eq (14)
loss.backward()                        # gradient reaches eps_theta's weights only
opt.step(); ema.update()               # Adam at 2e-4, EMA decay 0.9999

The settings from the paper's Appendix B: Adam at learning rate $2\times10^{-4}$ for CIFAR-10 ($2\times10^{-5}$ at $256\times256$), batch size 128 (64 at $256\times256$), an exponential moving average of the weights with decay 0.9999, dropout 0.1 on CIFAR-10, and random horizontal flips on every dataset except LSUN Bedroom. The CIFAR-10 model trains at 21 steps per second and reaches 800k steps in 10.6 hours on the TPU v3-8. Reported scores come from the checkpoint with the lowest FID during training.

# Algorithm 2: sampling (1000 sequential network calls)
x = randn(B, 3, 32, 32)                          # x_T ~ N(0, I)
for t in range(T, 0, -1):                        # t = 1000, 999, ..., 1
    e    = eps_theta(x, t)                       # predicted noise
    mean = (x - beta[t] / sqrt(1 - abar[t]) * e) / sqrt(alpha[t])  # eq (11)
    z    = randn_like(x) if t > 1 else 0         # no noise on the last step
    x    = mean + sqrt(sigma2[t]) * z            # sigma2 = beta or beta_tilde
return x                                         # x_0, a new image

The released TensorFlow sampler differs from this listing in one respect. It converts the predicted noise into $\hat{\mathbf{x}}_0$ with (15), clips $\hat{\mathbf{x}}_0$ to $[-1,1]$, and then takes the posterior mean (7) at the clipped value. When the clip changes nothing this is the same as (11); when the network predicts an out-of-range image, the code pulls it back into range and Algorithm 2 does not.

Every step above scales and adds real-valued vectors, so DDPM applies to continuous data. The paper uses only images: CIFAR-10 at $32\times32$, and CelebA-HQ and LSUN Bedroom, Church and Cat at $256\times256$. Discrete data such as text does not fit directly: adding 0.3 to token id 4017 does not give a token. Diffusion models for language diffuse a continuous embedding of the tokens or define a discrete corruption process instead; both are later work.

Results: sharp samples, average likelihood

On unconditional CIFAR-10 the $L_{\text{simple}}$ model reaches an Inception Score of 9.46 (a score for how confidently and how diversely an ImageNet classifier labels the samples; higher is better) and an FID of 3.17. The paper computes FID against the training set, as is standard; against the test set the score is 5.24. The paper reports 3.17 as state of the art for unconditional generation and says the model beats most models in the literature, including class-conditional ones. In its Table 1, unconditional StyleGAN2 with ADA scores 3.26; the class-conditional StyleGAN2 with ADA scores 2.67 and is better than DDPM.

At $256\times256$, the paper describes LSUN sample quality as similar to ProgressiveGAN. Its Table 3 gives FIDs of 6.36 on LSUN Bedroom (4.90 for a larger 256-million-parameter model), 7.89 on Church and 19.75 on Cat, against 8.34, 6.42 and 37.52 for ProgressiveGAN; StyleGAN2 scores 3.86 on Church and 6.93 on Cat. CelebA-HQ appears only as samples.

As a likelihood model DDPM is not competitive. Bits per dimension is the average number of bits needed to encode one pixel value under the model, so lower is better. At 3.75 bits per dimension (an upper bound, from (5)) it beats the estimates reported for energy-based models and score matching but trails the best autoregressive models and flows. The paper splits the codelength of its best model into a rate (bits transmitted by the reverse steps) of 1.78 bits/dim and a distortion of 1.97 bits/dim, where the distortion corresponds to an RMSE of 0.95 on the 0 to 255 pixel scale. More than half of the lossless codelength therefore describes differences too small to see. Its rate-distortion table shows where the bits go during sampling: after 900 of the 1000 reverse steps the rate is 0.12 bits/dim and the RMSE is 12.0; the last 100 steps add 1.66 bits/dim to bring the RMSE down to 0.95.

The 1000 sequential network calls set the agenda for the next few years. DDIM reused a trained DDPM with a sampler that skips steps, matching 1000-step quality in 20 to 100 steps. Dhariwal and Nichol improved the architecture and added classifier guidance, and classifier-free guidance removed the classifier. Latent diffusion ran the same process in the compressed latent space of an autoencoder to cut the cost of each step, which is the basis of Stable Diffusion. Song et al. recast the discrete chain as a stochastic differential equation, with DDPM's forward process as its variance-preserving case.