VerifiedarXiv:2403.1810329 min
Diffusion · Foundations

Tutorial on Diffusion Models for Imaging and Vision

Four derivations of the same diffusion model: as a variational autoencoder, as a chain of denoising steps, by score matching, and as a stochastic differential equation.

Stanley Chan's tutorial builds a variational autoencoder, a noise-predicting denoiser, a score model and a stochastic differential equation from first principles, and shows that all four train the same network and sample with the same reverse process.

Explaining the paperTutorial on Diffusion Models for Imaging and VisionStanley H. Chan · Purdue University · 2024 · arXiv:2403.18103 ↗

A standard diffusion model trains one network to look at an image with a known amount of Gaussian noise added and predict that noise. This page follows the tutorial's four derivations of that loss and of the sampler built on it, with ten figures you can drive.

A generative model for images has one job: produce new images that look like samples from the training set without copying any of them. In probability terms, the training images are draws from an unknown distribution p(x)p(\mathbf{x}) over all possible images, and the model must draw fresh samples from it. A 64×64 color image is a vector of 12,288 numbers, and you hold only a finite set of examples, never the density itself.

Diffusion models work from the opposite direction. Destroying an image is easy: add a little Gaussian noise, then a little more, and after about a thousand steps any image is indistinguishable from pure noise. That direction needs no learning. A network is trained to undo one small step of it, and generation starts from pure noise and applies the learned undo-step a thousand times.

Chan's tutorial (Purdue, 2024; three arXiv versions, the latest January 2025) derives this machine four times, in dependency order:

Section numbers and equation numbers below refer to the current version (v3) unless marked v1. The page renumbers the equations it uses as (1) to (17).

How a VAE is trained: the ELBO

A VAE has two networks. The encoder maps an image x\mathbf{x} to a short latent code z\mathbf{z}; the decoder maps a code back to an image. Training also pushes the codes toward a fixed distribution that is easy to sample, the standard Gaussian p(z)=N(0,I)p(\mathbf{z}) = \mathcal{N}(\mathbf{0},\mathbf{I}). Generation then takes two lines: draw z\mathbf{z} from the Gaussian and decode it.

Both networks output distributions. The encoder qϕ(z∣x)q_{\boldsymbol{\phi}}(\mathbf{z}\mid\mathbf{x}) is a Gaussian whose mean and per-coordinate variance come from a network with weights ϕ\boldsymbol{\phi}. The decoder pθ(x∣z)p_{\boldsymbol{\theta}}(\mathbf{x}\mid\mathbf{z}) is a Gaussian centered on the network's reconstruction. A diagonal Gaussian is the default because its parameters are two vectors a network can output, it is sampled in one line, and its KL divergence to the standard Gaussian has a closed form.

The natural objective is the log-likelihood log⁡p(x)\log p(\mathbf{x}) of real images, but it contains an integral over every possible code and cannot be computed. The tutorial splits it into a computable term and a remainder:

log⁡p(x)=Eqϕ(z∣x) ⁣[log⁡p(x,z)qϕ(z∣x)]⏟ELBO(x)+DKL ⁣(qϕ(z∣x) ∥ p(z∣x))\log p(\mathbf{x}) = \underbrace{\mathbb{E}_{q_{\boldsymbol{\phi}}(\mathbf{z}\mid\mathbf{x})}\!\left[\log \frac{p(\mathbf{x},\mathbf{z})}{q_{\boldsymbol{\phi}}(\mathbf{z}\mid\mathbf{x})}\right]}_{\text{ELBO}(\mathbf{x})} + \mathbb{D}_{\text{KL}}\!\big(q_{\boldsymbol{\phi}}(\mathbf{z}\mid\mathbf{x}) \,\|\, p(\mathbf{z}\mid\mathbf{x})\big)log⁡p(x)=Eqϕ(z∣x) ⁣[log⁡p(x,z)qϕ(z∣x)]⏟ELBO(x)+DKL ⁣(qϕ(z∣x) ∥ p(z∣x))\begin{aligned} \log p(\mathbf{x}) &= \underbrace{\mathbb{E}_{q_{\boldsymbol{\phi}}(\mathbf{z}\mid\mathbf{x})}\!\left[\log \frac{p(\mathbf{x},\mathbf{z})}{q_{\boldsymbol{\phi}}(\mathbf{z}\mid\mathbf{x})}\right]}_{\text{ELBO}(\mathbf{x})} \\ &\quad + \mathbb{D}_{\text{KL}}\!\big(q_{\boldsymbol{\phi}}(\mathbf{z}\mid\mathbf{x}) \,\|\, p(\mathbf{z}\mid\mathbf{x})\big) \end{aligned}
(1)

The second term is a Kullback-Leibler (KL) divergence, a measure of how far one distribution is from another, and it is never negative. So the first term, the evidence lower bound, can never exceed log⁡p(x)\log p(\mathbf{x}). The KL in (1) is the gap between the bound and the truth, and it compares the encoder with the true posterior p(z∣x)p(\mathbf{z}\mid\mathbf{x}), a distribution nobody can compute. When the encoder equals that posterior, the gap is zero and the bound is exact.

Figure 1 shows the gap for a one-dimensional latent. The slider moves the encoder's Gaussian onto the true posterior; the bar below splits log⁡p(x)\log p(\mathbf{x}) into the ELBO and the KL gap.

Figure 1 · the evidence lower bound
28%
Top: the true posterior (amber) and the encoder's Gaussian q(z | x). Bottom: the full bar is log⁡p(x)\log p(\mathbf{x}), split into the ELBO and the KL gap. Drag match q to 100% and the gap reaches zero.

Since log⁡p(x)\log p(\mathbf{x}) does not depend on ϕ\boldsymbol{\phi}, raising the ELBO by moving the encoder is the same as shrinking the gap. Rearranged, the ELBO splits into the two jobs the VAE does:

ELBO(x)=Eqϕ(z∣x)[log⁡pθ(x∣z)]⏟reconstruction−DKL(qϕ(z∣x) ∥ p(z))⏟prior matching\text{ELBO}(\mathbf{x}) = \underbrace{\mathbb{E}_{q_{\boldsymbol{\phi}}(\mathbf{z}\mid\mathbf{x})}\big[\log p_{\boldsymbol{\theta}}(\mathbf{x}\mid\mathbf{z})\big]}_{\text{reconstruction}} - \underbrace{\mathbb{D}_{\text{KL}}\big(q_{\boldsymbol{\phi}}(\mathbf{z}\mid\mathbf{x}) \,\|\, p(\mathbf{z})\big)}_{\text{prior matching}}ELBO(x)=Eqϕ(z∣x)[log⁡pθ(x∣z)]⏟reconstruction−DKL(qϕ(z∣x) ∥ p(z))⏟prior matching\begin{aligned} \text{ELBO}(\mathbf{x}) &= \underbrace{\mathbb{E}_{q_{\boldsymbol{\phi}}(\mathbf{z}\mid\mathbf{x})}\big[\log p_{\boldsymbol{\theta}}(\mathbf{x}\mid\mathbf{z})\big]}_{\text{reconstruction}} \\ &\quad - \underbrace{\mathbb{D}_{\text{KL}}\big(q_{\boldsymbol{\phi}}(\mathbf{z}\mid\mathbf{x}) \,\|\, p(\mathbf{z})\big)}_{\text{prior matching}} \end{aligned}
(2)

The reconstruction term rewards the decoder for rebuilding x\mathbf{x} from its code; with a Gaussian decoder it is a scaled squared error. The prior-matching term penalizes codes that stray from N(0,I)\mathcal{N}(\mathbf{0},\mathbf{I}), which keeps the latent space samplable. This KL is against the prior p(z)p(\mathbf{z}), a different term from the gap in (1), which is against the posterior. The two terms of (2) pull in opposite directions: maximizing the reconstruction term alone favors a distinct, precise code per image, and minimizing the prior KL alone pushes every image's code distribution toward the same Gaussian.

Training needs the gradient of the ELBO with respect to ϕ\boldsymbol{\phi}, but ϕ\boldsymbol{\phi} enters through a sampling step, and a random draw has no derivative with respect to the parameters of its distribution. The reparameterization trick (Kingma and Welling) moves the randomness into a parameter-free input:

z=μϕ(x)+σϕ(x)⊙ϵ,ϵ∼N(0,I)\mathbf{z} = \boldsymbol{\mu}_{\boldsymbol{\phi}}(\mathbf{x}) + \boldsymbol{\sigma}_{\boldsymbol{\phi}}(\mathbf{x}) \odot \boldsymbol{\epsilon}, \qquad \boldsymbol{\epsilon} \sim \mathcal{N}(\mathbf{0},\mathbf{I})
(3)

Now ϵ\boldsymbol{\epsilon} carries the randomness, and ϕ\boldsymbol{\phi} appears only in a multiply and an add, which backpropagation handles. The alternative, the score-function estimator, differentiates the sampling distribution directly; Kingma and Welling found its gradient "exhibits very high variance" and called it impractical for their purposes. The form of (3), a clean value scaled plus standard noise scaled, is the form the diffusion forward process takes in the next section.

The forward noising process

DDPM (denoising diffusion probabilistic models, Ho, Jain and Abbeel, 2020) changes the VAE in two ways. The latent is the same size as the image: it is the image with noise added. And there is a chain of TT latents, x1\mathbf{x}_1 to xT\mathbf{x}_T, each noisier than the last, with x0\mathbf{x}_0 the clean image. The "encoder" has no weights; it adds noise on a fixed schedule:

q(xt∣xt−1)=N ⁣(xt ; αt xt−1,  (1−αt) I)q(\mathbf{x}_t \mid \mathbf{x}_{t-1}) = \mathcal{N}\!\big(\mathbf{x}_t \,;\, \sqrt{\alpha_t}\,\mathbf{x}_{t-1},\; (1-\alpha_t)\,\mathbf{I}\big)
(4)

The per-step scale αt\alpha_t is a number just below 1. Ho's schedule writes βt=1−αt\beta_t = 1-\alpha_t and raises it linearly from 10−410^{-4} at t=1t=1 to 0.02 at t=T=1000t=T=1000. The signal is shrunk by αt\sqrt{\alpha_t} and the added noise has variance 1−αt1-\alpha_t, and these two are tied: if xt−1\mathbf{x}_{t-1} has unit variance, then xt\mathbf{x}_t has variance αt+(1−αt)=1\alpha_t + (1-\alpha_t) = 1. The tutorial picks αt\sqrt{\alpha_t} so that the variance "will not explode and vanish after many iterations".1 A step with signal scale aa and noise scale bb repeated forever settles at variance b2/(1−a2)b^2/(1-a^2), and b=1−a2b=\sqrt{1-a^2} makes that 1.

Reaching x400\mathbf{x}_{400} does not require 400 sequential steps. Each step is a linear map plus independent Gaussian noise, and a sum of independent Gaussians is Gaussian with the variances added. Composing tt steps therefore gives a scaled copy of x0\mathbf{x}_0 plus one Gaussian:

xt=αˉt x0+1−αˉt ϵ,αˉt=∏i=1tαi\mathbf{x}_t = \sqrt{\bar{\alpha}_t}\,\mathbf{x}_0 + \sqrt{1-\bar{\alpha}_t}\,\boldsymbol{\epsilon}, \qquad \bar{\alpha}_t = \prod_{i=1}^{t}\alpha_ixt=αˉt x0+1−αˉt ϵ,αˉt=∏i=1tαi\begin{gathered} \mathbf{x}_t = \sqrt{\bar{\alpha}_t}\,\mathbf{x}_0 + \sqrt{1-\bar{\alpha}_t}\,\boldsymbol{\epsilon}, \\ \bar{\alpha}_t = \prod_{i=1}^{t}\alpha_i \end{gathered}
(5)

with ϵ∼N(0,I)\boldsymbol{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I}). The cumulative product αˉt\bar{\alpha}_t says how far along the trip you are. On Ho's schedule, αˉt\bar{\alpha}_t crosses 0.5 near t=259t=259, where signal and noise have equal weight (0.707 each). At t=400t=400, αˉt=0.195\bar{\alpha}_t = 0.195 and

x400=0.442 x0+0.897 ϵ.\mathbf{x}_{400} = 0.442\,\mathbf{x}_0 + 0.897\,\boldsymbol{\epsilon}.

At t=1000t=1000, αˉt≈0.0064\sqrt{\bar{\alpha}_t} \approx 0.0064: what is left is a standard Gaussian with a trace of the image. Equation (5) has the same shape as the reparameterization (3), with αˉt x0\sqrt{\bar\alpha_t}\,\mathbf{x}_0 in place of the encoder mean and 1−αˉt\sqrt{1-\bar\alpha_t} in place of its standard deviation.

Figure 2 · the forward process
t = 1
Drag step t. The clean point cloud (amber) is scaled by αˉt\sqrt{\bar{\alpha}_t} while noise of variance 1−αˉt1-\bar{\alpha}_t fills in; the readout shows both scalars. Every frame is computed in one jump from (5), with no stepping.

How DDPM is trained: from the ELBO to predicting the noise

The model is the reverse direction: a network that takes xt\mathbf{x}_t and produces the distribution of the slightly cleaner xt−1\mathbf{x}_{t-1}. Each reverse step is a Gaussian  pθ(xt−1∣xt)\;p_{\boldsymbol{\theta}}(\mathbf{x}_{t-1}\mid\mathbf{x}_t) whose mean μθ(xt)\boldsymbol{\mu}_{\boldsymbol{\theta}}(\mathbf{x}_t) a network predicts, a small decoder. The chain is trained by the ELBO, as the VAE was. With the chain in place of a single latent, the tutorial's Theorem 2.4 writes it as three kinds of term:

ELBO=E[log⁡pθ(x0∣x1)]−DKL(q(xT∣x0) ∥ p(xT))−∑t=2TE[DKL(q(xt−1∣xt,x0) ∥ pθ(xt−1∣xt))]\text{ELBO} = \mathbb{E}\big[\log p_{\boldsymbol{\theta}}(\mathbf{x}_0\mid\mathbf{x}_1)\big] - \mathbb{D}_{\text{KL}}\big(q(\mathbf{x}_T\mid\mathbf{x}_0)\,\|\,p(\mathbf{x}_T)\big) - \sum_{t=2}^{T}\mathbb{E}\Big[\mathbb{D}_{\text{KL}}\big(q(\mathbf{x}_{t-1}\mid\mathbf{x}_t,\mathbf{x}_0)\,\|\,p_{\boldsymbol{\theta}}(\mathbf{x}_{t-1}\mid\mathbf{x}_t)\big)\Big]ELBO=E[log⁡pθ(x0∣x1)]−DKL(q(xT∣x0) ∥ p(xT))−∑t=2TE[DKL(q(xt−1∣xt,x0) ∥ pθ(xt−1∣xt))]\begin{aligned} \text{ELBO} &= \mathbb{E}\big[\log p_{\boldsymbol{\theta}}(\mathbf{x}_0\mid\mathbf{x}_1)\big] \\ &\quad - \mathbb{D}_{\text{KL}}\big(q(\mathbf{x}_T\mid\mathbf{x}_0)\,\|\,p(\mathbf{x}_T)\big) \\ &\quad - \sum_{t=2}^{T}\mathbb{E}\Big[\mathbb{D}_{\text{KL}}\big(q(\mathbf{x}_{t-1}\mid\mathbf{x}_t,\mathbf{x}_0)\,\|\,p_{\boldsymbol{\theta}}(\mathbf{x}_{t-1}\mid\mathbf{x}_t)\big)\Big] \end{aligned}
(6)

The first term is reconstruction at the last step, the second is prior matching at xT\mathbf{x}_T (no trainable weights, since the forward process is fixed), and the sum is what the tutorial calls consistency: at every level, the model's reverse step should agree with the true reverse step. The true reverse step q(xt−1∣xt,x0)q(\mathbf{x}_{t-1}\mid\mathbf{x}_t,\mathbf{x}_0) is conditioned on the clean image, and with x0\mathbf{x}_0 known it is a Gaussian with a closed form (Theorem 2.5). Its mean is a linear combination of where the sample is and where it started:

μq(xt,x0)=αt (1−αˉt−1)1−αˉt xt+αˉt−1 (1−αt)1−αˉt x0\boldsymbol{\mu}_q(\mathbf{x}_t,\mathbf{x}_0) = \frac{\sqrt{\alpha_t}\,(1-\bar{\alpha}_{t-1})}{1-\bar{\alpha}_t}\,\mathbf{x}_t + \frac{\sqrt{\bar{\alpha}_{t-1}}\,(1-\alpha_t)}{1-\bar{\alpha}_t}\,\mathbf{x}_0μq(xt,x0)=αt (1−αˉt−1)1−αˉt xt+αˉt−1 (1−αt)1−αˉt x0\begin{aligned} \boldsymbol{\mu}_q(\mathbf{x}_t,\mathbf{x}_0) &= \frac{\sqrt{\alpha_t}\,(1-\bar{\alpha}_{t-1})}{1-\bar{\alpha}_t}\,\mathbf{x}_t \\ &\quad + \frac{\sqrt{\bar{\alpha}_{t-1}}\,(1-\alpha_t)}{1-\bar{\alpha}_t}\,\mathbf{x}_0 \end{aligned}
(7)

and its variance is

σq2(t)=(1−αt)(1−αˉt−1)1−αˉt\sigma_q^2(t) = \frac{(1-\alpha_t)(1-\bar{\alpha}_{t-1})}{1-\bar{\alpha}_t}
(8)

This is Ho's β~t\tilde{\beta}_t. The tutorial's Eq 2.26 prints a square root on αˉt−1\bar{\alpha}_{t-1} in the numerator; its own derivation (Eq 2.32) and Ho's paper have none, and (8) follows them. The printed form would give half the correct variance at small tt (ratio 0.500 at t=10t=10, 0.69 at t=400t=400).

The model's reverse step uses the same variance σq2(t)\sigma_q^2(t), so only the means can differ. The KL between two Gaussians with equal variance is the squared distance between their means divided by 2σq2(t)2\sigma_q^2(t), and each consistency term becomes a regression of μθ(xt)\boldsymbol{\mu}_{\boldsymbol{\theta}}(\mathbf{x}_t) onto μq\boldsymbol{\mu}_q. The target μq\boldsymbol{\mu}_q depends on x0\mathbf{x}_0, and (5) solves for x0\mathbf{x}_0 in terms of xt\mathbf{x}_t and the noise:

x0=xt−1−αˉt ϵαˉt\mathbf{x}_0 = \frac{\mathbf{x}_t - \sqrt{1-\bar\alpha_t}\,\boldsymbol{\epsilon}}{\sqrt{\bar\alpha_t}}

Substituting into (7) turns the target into a function of xt\mathbf{x}_t and ϵ\boldsymbol{\epsilon}, and the tutorial chooses the network's mean to have the same form, with a network ϵ^θ\hat{\boldsymbol{\epsilon}}_{\boldsymbol{\theta}} in place of the true noise:

μθ(xt)=1αt(xt−1−αt1−αˉt ϵ^θ(xt,t))\boldsymbol{\mu}_{\boldsymbol{\theta}}(\mathbf{x}_t) = \frac{1}{\sqrt{\alpha_t}}\left(\mathbf{x}_t - \frac{1-\alpha_t}{\sqrt{1-\bar{\alpha}_t}}\,\hat{\boldsymbol{\epsilon}}_{\boldsymbol{\theta}}(\mathbf{x}_t,t)\right)
(9)

The per-step scale αt\sqrt{\alpha_t} sits outside and the cumulative 1−αˉt\sqrt{1-\bar\alpha_t} sits under the noise estimate. The difference of the two means is now a multiple of ϵ^θ−ϵ\hat{\boldsymbol{\epsilon}}_{\boldsymbol{\theta}} - \boldsymbol{\epsilon}, and the consistency term at level tt becomes

wt ∥ϵ^θ(xt,t)−ϵ∥2,wt=(1−αt)22σq2(t) (1−αˉt) αtw_t\,\big\lVert \hat{\boldsymbol{\epsilon}}_{\boldsymbol{\theta}}(\mathbf{x}_t,t) - \boldsymbol{\epsilon}\big\rVert^2, \qquad w_t = \frac{(1-\alpha_t)^2}{2\sigma_q^2(t)\,(1-\bar{\alpha}_t)\,\alpha_t}wt ∥ϵ^θ(xt,t)−ϵ∥2,wt=(1−αt)22σq2(t) (1−αˉt) αt\begin{gathered} w_t\,\big\lVert \hat{\boldsymbol{\epsilon}}_{\boldsymbol{\theta}}(\mathbf{x}_t,t) - \boldsymbol{\epsilon}\big\rVert^2, \\ w_t = \frac{(1-\alpha_t)^2}{2\sigma_q^2(t)\,(1-\bar{\alpha}_t)\,\alpha_t} \end{gathered}

The training procedure the tutorial writes out drops wtw_t and samples tt uniformly, which gives the DDPM loss:

L(θ)=Ex0, t, ϵ[∥ ϵ^θ(αˉt x0+1−αˉt ϵ,  t)−ϵ ∥2]\mathcal{L}(\boldsymbol{\theta}) = \mathbb{E}_{\mathbf{x}_0,\,t,\,\boldsymbol{\epsilon}}\Big[\big\lVert\, \hat{\boldsymbol{\epsilon}}_{\boldsymbol{\theta}}(\sqrt{\bar{\alpha}_t}\,\mathbf{x}_0 + \sqrt{1-\bar{\alpha}_t}\,\boldsymbol{\epsilon},\; t) - \boldsymbol{\epsilon} \,\big\rVert^2\Big]L(θ)=Ex0, t, ϵ[∥ ϵ^θ(αˉt x0+1−αˉt ϵ,  t)−ϵ ∥2]\begin{aligned} \mathcal{L}(\boldsymbol{\theta}) = \mathbb{E}_{\mathbf{x}_0,\,t,\,\boldsymbol{\epsilon}}\Big[\big\lVert\, &\hat{\boldsymbol{\epsilon}}_{\boldsymbol{\theta}}(\sqrt{\bar{\alpha}_t}\,\mathbf{x}_0 + \sqrt{1-\bar{\alpha}_t}\,\boldsymbol{\epsilon},\; t) \\ &- \boldsymbol{\epsilon} \,\big\rVert^2\Big] \end{aligned}
(10)

With the weights dropped, (10) is no longer a bound on the log-likelihood; it is a reweighted sum of the same terms, and the reweighting changes which noise levels the network practices on. With σq2\sigma_q^2 from (8), wtw_t simplifies to (1−αt)/(2αt(1−αˉt−1))(1-\alpha_t)/(2\alpha_t(1-\bar\alpha_{t-1})): 0.600 at t=2t=2, 0.0103 at t=100t=100, a minimum of 0.00498 at t=350t=350, and 0.0102 at t=1000t=1000, a factor of 59 from end to end. Under the bound, steps 2 to 100 carry 39.2% of the total weight; under (10) they carry 9.9%, their share of the steps. At t=2t=2 the noise standard deviation is 0.015 on a pixel scale of [−1, 1], about two of the 256 intensity levels. Ho et al. explain the choice this way: the small-tt terms train the network to remove very small amounts of noise, and down-weighting them lets it focus on the harder denoising at larger tt. They report better sample quality with the unweighted loss. Figure 3 plots the cumulative weight; pick the σₜ² = β̃ₜ tab for the tutorial's variance (8).

Figure 3 · where the loss puts its weight
c = 100
Cumulative share of the loss weight on steps 2 through cutoff c. The variational bound rises steeply at small t; the uniform loss (10), labeled L_simple as in Ho's paper, is the diagonal. The β̃ₜ tab uses the tutorial's σ_q²; the βₜ tab uses Ho's other choice of sampling variance.

Given xt\mathbf{x}_t, the clean image, the noise, and the posterior mean (7) determine one another through (5), so a network that predicts any one of them can be converted to the other two. The tutorial motivates the noise target by analogy with residual denoisers in image processing, which predict the noise rather than the signal. Ho and coauthors also tried predicting x0\mathbf{x}_0 and found it led to worse sample quality early in their experiments.

A concrete step on 64×64 color images with Ho's schedule: draw a batch of clean images, shape [B, 3, 64, 64]; draw a level per image, say t=400t=400; draw ϵ\boldsymbol{\epsilon} of the same shape; scale and add them as in (5), with coefficients 0.442 and 0.897; run the network on (x400,400)(\mathbf{x}_{400}, 400) to get ϵ^\hat{\boldsymbol{\epsilon}} of shape [B, 3, 64, 64]; take the mean squared error against ϵ\boldsymbol{\epsilon}. The gradient flows into the network's weights only; there is no chain to backpropagate through, because (5) jumps straight to level 400. One network, given tt as an input, serves all 1000 levels.

# one DDPM training step (Ho's linear schedule, T = 1000)
x0   = sample_batch()                   # clean images, shape [B, 3, 64, 64]
t    = randint(1, T + 1, size=B)        # one noise level per image, e.g. 400
eps  = randn_like(x0)                   # the target
xt   = sqrt(abar[t]) * x0 + sqrt(1 - abar[t]) * eps   # t=400: 0.442, 0.897
loss = ((eps_hat(xt, t) - eps) ** 2).mean()            # eq (10), weights dropped
loss.backward()                         # gradient reaches eps_hat's weights only
opt.step()

Sampling from DDPM

Generation runs the chain backward. Start from xT∼N(0,I)\mathbf{x}_T \sim \mathcal{N}(\mathbf{0},\mathbf{I}); at each level, compute the mean (9) with the trained network and add fresh Gaussian noise of standard deviation σq(t)\sigma_q(t):

xt−1=1αt(xt−1−αt1−αˉt ϵ^θ(xt,t))+σq(t) z,z∼N(0,I)\mathbf{x}_{t-1} = \frac{1}{\sqrt{\alpha_t}}\left(\mathbf{x}_t - \frac{1-\alpha_t}{\sqrt{1-\bar{\alpha}_t}}\,\hat{\boldsymbol{\epsilon}}_{\boldsymbol{\theta}}(\mathbf{x}_t,t)\right) + \sigma_q(t)\,\mathbf{z}, \qquad \mathbf{z}\sim\mathcal{N}(\mathbf{0},\mathbf{I})xt−1=1αt(xt−1−αt1−αˉt ϵ^θ(xt,t))+σq(t) z,z∼N(0,I)\begin{aligned} \mathbf{x}_{t-1} &= \frac{1}{\sqrt{\alpha_t}}\left(\mathbf{x}_t - \frac{1-\alpha_t}{\sqrt{1-\bar{\alpha}_t}}\,\hat{\boldsymbol{\epsilon}}_{\boldsymbol{\theta}}(\mathbf{x}_t,t)\right) \\ &\quad + \sigma_q(t)\,\mathbf{z}, \qquad \mathbf{z}\sim\mathcal{N}(\mathbf{0},\mathbf{I}) \end{aligned}
(11)

At t=400t=400 on Ho's schedule this reads

x399=1.0040 (x400−0.00897 ϵ^)+0.0896 z.\mathbf{x}_{399} = 1.0040\,(\mathbf{x}_{400} - 0.00897\,\hat{\boldsymbol{\epsilon}}) + 0.0896\,\mathbf{z}.

The step subtracts 0.00897 times the predicted noise, scales up by 1.0040 to undo the forward shrink of step 400, and adds fresh noise with standard deviation 0.0896. At t=1t=1, σq2(1)=0\sigma_q^2(1) = 0 because αˉ0=1\bar\alpha_0 = 1, so the last step adds no noise and returns the mean. The reverse process is stochastic at every other step; two runs from the same xT\mathbf{x}_T end at different images.

# DDPM sampling, eq (11): T network calls per image
x = randn(3, 64, 64)                    # x_T, pure Gaussian noise
for t in range(T, 0, -1):
    e    = eps_hat(x, t)
    mean = (x - (1 - a[t]) / sqrt(1 - abar[t]) * e) / sqrt(a[t])
    z    = randn_like(x) if t > 1 else 0
    x    = mean + sigma_q[t] * z        # sigma_q[t]^2 from eq (8)
return x                                # t=400: 1.0040*(x - 0.00897 e) + 0.0896 z

Figure 4 runs (11) on six points in the plane, arranged as two groups of three. In place of a trained network it uses the exact optimal noise predictor for this dataset, so any failure comes from the sampler alone. Play it with 1000 steps and every particle ends on a data point. Then try 10, 2 and 1 steps, and the no noise tab, which drops the σq(t) z\sigma_q(t)\,\mathbf{z} term.

Figure 4 · DDPM sampling, step by step
100%
T = 1000
Press Play or drag the scrubber. Particles start as Gaussian noise and end on the data points; the readout reports how many land on one. The β̃ₜ tab is the tutorial's variance (8). With one step every particle lands in the middle; with no added noise they collapse onto the two inner points. show x̂₀ draws each particle's current estimate of the clean point.

With no injected noise, every particle follows the mean, and the means pull toward the average of the data points each particle could still become. The noise keeps particles spread so that they reach the outer points too. With one step, the sampler can only output the posterior mean of x0\mathbf{x}_0 given pure noise, which is the center of the data.

The cost is the step count: 1000 network calls per image. The tutorial quotes Song et al.: a DDPM takes more than 1000 hours to generate 50,000 images of size 256×256 on a standard GPU. Section 2.6 of the tutorial (absent from v1) covers DDIM, which replaces the Markov chain with a non-Markovian one that skips levels and has a free noise scale σt\sigma_t in its update (Eq 2.59). That update's only random term is σtϵt\sigma_t\boldsymbol{\epsilon}_t, so σt=0\sigma_t = 0 makes sampling deterministic. DDIM reuses a network trained with (10) unchanged. In that section the tutorial switches to DDIM's notation, where αt\alpha_t means the cumulative product; this page keeps αˉt\bar\alpha_t for it throughout.

The score and Langevin dynamics

The tutorial's third chapter starts again from a different question. Given a density p(x)p(\mathbf{x}), how do you draw samples from it? Score-matching Langevin dynamics (SMLD), from Song and Ermon's NCSN paper (2019), answers with two pieces, and neither mentions a noising chain.

The first piece is the score of the density:

s(x)=∇xlog⁡p(x)\mathbf{s}(\mathbf{x}) = \nabla_{\mathbf{x}} \log p(\mathbf{x})
(12)

At every point x\mathbf{x} it is a vector pointing in the direction in which the log-density rises fastest. For a one-dimensional Gaussian N(μ,s2)\mathcal{N}(\mu, s^2), the score is −(x−μ)/s2-(x-\mu)/s^2: at x=μ+2sx = \mu + 2s it is −2/s-2/s, pointing back toward the mean and longer the farther out you are. The gradient is with respect to the data point, which the tutorial calls the Stein score, as distinct from the score ∇θlog⁡pθ(x)\nabla_{\boldsymbol{\theta}}\log p_{\boldsymbol{\theta}}(\mathbf{x}) used in maximum-likelihood estimation.2

The score does not depend on the normalizing constant. Any density can be written p(x)=e−U(x)/Zp(\mathbf{x}) = e^{-U(\mathbf{x})}/Z, where ZZ is the integral that makes it sum to one over all 12,288-dimensional images and cannot be computed. Taking the log turns the division into a subtraction of log⁡Z\log Z, a constant, and its gradient with respect to x\mathbf{x} is zero, so ∇xlog⁡p=−∇xU\nabla_{\mathbf{x}}\log p = -\nabla_{\mathbf{x}}U, and a model of the score never has to estimate ZZ.

Figure 5 draws the score of a mixture of six Gaussian clusters after blurring by noise of level σ\sigma. At small σ\sigma each arrow points to its nearest cluster; at large σ\sigma the clusters merge into one broad hill and the arrows point toward its center.

Figure 5 · the score field
σ = 1.00
Arrows are the score ∇log⁡pσ(x)\nabla\log p_\sigma(\mathbf{x}) of the blurred mixture; dots are the cluster centers. Drag noise σ from 0.25 to 3.5.

The second piece turns the score into a sampler. Langevin dynamics takes a small step along the score and adds Gaussian noise (the tutorial's Definition 3.1):

xt+1=xt+τ ∇xlog⁡p(xt)+2τ z,z∼N(0,I)\mathbf{x}_{t+1} = \mathbf{x}_t + \tau\,\nabla_{\mathbf{x}}\log p(\mathbf{x}_t) + \sqrt{2\tau}\,\mathbf{z}, \qquad \mathbf{z}\sim\mathcal{N}(\mathbf{0},\mathbf{I})xt+1=xt+τ ∇xlog⁡p(xt)+2τ z,z∼N(0,I)\begin{gathered} \mathbf{x}_{t+1} = \mathbf{x}_t + \tau\,\nabla_{\mathbf{x}}\log p(\mathbf{x}_t) + \sqrt{2\tau}\,\mathbf{z}, \\ \mathbf{z}\sim\mathcal{N}(\mathbf{0},\mathbf{I}) \end{gathered}
(13)

The sign is a plus because the step climbs the log-density. Without the noise term, (13) is gradient ascent: every particle climbs to the nearest peak and stops, which finds modes, not samples. With the noise, particles keep moving around the high-density regions, and for a small step size τ\tau and many iterations the distribution of a particle approaches pp. The tutorial summarizes it as "gradient descent + noise". The 2τ\sqrt{2\tau} is not a free choice: Figure 6 scales the noise term by a multiplier kk, and only near k=1k=1 does the cloud match the data.

Figure 6 · Langevin dynamics
k = 1.00
Particles follow the score toward the data with the noise term of (13) scaled by k. At k=0k=0 they collapse onto the peaks; near k=1k=1 they spread across every cluster and keep moving; at k=1.8k=1.8 the noise wins and they scatter past the clusters.

Denoising score matching

Langevin needs the score of the data distribution, and nobody has it: there are training images and no density. A regression needs targets, and the true score cannot be evaluated at any point. Denoising score matching (Vincent, 2011) supplies a target by adding noise of known size. Perturb a clean image x′\mathbf{x}' to x=x′+σz\mathbf{x} = \mathbf{x}' + \sigma\mathbf{z} with z∼N(0,I)\mathbf{z}\sim\mathcal{N}(\mathbf{0},\mathbf{I}). The score of that single Gaussian kernel is known exactly:

∇xlog⁡q(x∣x′)=−x−x′σ2=−σzσ2=−zσ\nabla_{\mathbf{x}}\log q(\mathbf{x}\mid\mathbf{x}') = -\frac{\mathbf{x}-\mathbf{x}'}{\sigma^2} = -\frac{\sigma\mathbf{z}}{\sigma^2} = -\frac{\mathbf{z}}{\sigma}

The 1/σ21/\sigma^2 belongs to the raw difference x−x′\mathbf{x}-\mathbf{x}', and the 1/σ1/\sigma to the unit noise z\mathbf{z}; they are the same quantity in two variables. With σ=0.1\sigma = 0.1 and z=(0.5,−1.2)\mathbf{z} = (0.5, -1.2), the offset is x−x′=(0.05,−0.12)\mathbf{x}-\mathbf{x}' = (0.05, -0.12), and both forms give the target (−5,12)(-5, 12). A network sθ\mathbf{s}_{\boldsymbol{\theta}} is trained to output it:

J(θ)=E[12∥sθ(x′+σz)+zσ∥2]J(\boldsymbol{\theta}) = \mathbb{E}\left[\tfrac{1}{2}\left\lVert \mathbf{s}_{\boldsymbol{\theta}}(\mathbf{x}' + \sigma\mathbf{z}) + \frac{\mathbf{z}}{\sigma} \right\rVert^2\right]
(14)

A single target −z/σ-\mathbf{z}/\sigma points at one particular clean image and is noisy. Many clean images could have produced the same x\mathbf{x}, and the least-squares fit averages their targets. Vincent proved that this average is the score of the noised distribution pσ=p∗N(0,σ2I)p_\sigma = p * \mathcal{N}(\mathbf{0},\sigma^2\mathbf{I}), so minimizing (14) trains sθ\mathbf{s}_{\boldsymbol{\theta}} toward ∇log⁡pσ\nabla\log p_\sigma without ever evaluating it.3 The tutorial puts it as "the score function sθs_\theta is supposed to take this noisy image and predict the noise z/σz/\sigma".

Compare (14) with DDPM's (10). Both add Gaussian noise to a clean sample and regress on that noise; (14) asks for −z/σ-\mathbf{z}/\sigma where (10) asks for ϵ\boldsymbol{\epsilon}. In DDPM's variables the noise on xt\mathbf{x}_t has standard deviation 1−αˉt\sqrt{1-\bar\alpha_t}, and the kernel score is −ϵ/1−αˉt-\boldsymbol{\epsilon}/\sqrt{1-\bar\alpha_t}. A trained noise predictor therefore gives a score estimate directly:

sθ(xt,t)=−ϵ^θ(xt,t)1−αˉt\mathbf{s}_{\boldsymbol{\theta}}(\mathbf{x}_t, t) = -\frac{\hat{\boldsymbol{\epsilon}}_{\boldsymbol{\theta}}(\mathbf{x}_t, t)}{\sqrt{1-\bar{\alpha}_t}}

One noise level is not enough for sampling. A score fit at a small σ\sigma is accurate only near the data, and Langevin started from noise spends its early steps far from the data, where training samples at that σ\sigma never land. Song and Ermon's noise-conditional score network (NCSN) gives the network σ\sigma as an input and trains it on a ladder of levels σ1>⋯>σL\sigma_1 > \dots > \sigma_L, with level σ\sigma's loss (14) weighted by λ(σ)\lambda(\sigma), chosen as σ2\sigma^2 "based on empirical findings". That weight turns (14) into DDPM's loss: multiplying (14) by σ2\sigma^2 gives

σ2⋅12∥sθ+zσ∥2=12∥σsθ+z∥2,\sigma^2 \cdot \tfrac{1}{2}\Big\lVert \mathbf{s}_{\boldsymbol{\theta}} + \frac{\mathbf{z}}{\sigma}\Big\rVert^2 = \tfrac{1}{2}\big\lVert \sigma\mathbf{s}_{\boldsymbol{\theta}} + \mathbf{z}\big\rVert^2,

which is (10)'s squared error, halved, with the noise prediction ϵ^=−σsθ\hat{\boldsymbol{\epsilon}} = -\sigma\mathbf{s}_{\boldsymbol{\theta}}. Sampling, called annealed Langevin, runs (13) at σ1\sigma_1, then at σ2\sigma_2, and so on down to the smallest level.

Why a denoiser computes the score: Tweedie's formula

DDPM trains a denoiser and SMLD trains a score model, and the previous section showed their losses coincide. A classical identity explains why the trained networks coincide too: Tweedie's formula, which the tutorial uses without naming.4

Take a clean signal x\mathbf{x} from any distribution, add Gaussian noise to get z=x+σϵ\mathbf{z} = \mathbf{x} + \sigma\boldsymbol{\epsilon}, and ask for the estimate D(z)D(\mathbf{z}) of x\mathbf{x} that minimizes the expected squared error E∥D(z)−x∥2\mathbb{E}\lVert D(\mathbf{z}) - \mathbf{x}\rVert^2. The minimizer is the posterior mean E[x∣z]\mathbb{E}[\mathbf{x}\mid\mathbf{z}], the average of all clean signals weighted by how likely each is to have produced z\mathbf{z}, because for any random variable the constant closest to it in mean squared error is its mean. (Version 1 of the tutorial, §2.9, skips this step as "some magical step" and points to a probability textbook; that section was removed in v2.) Tweedie's formula relates this denoiser to the score of the noised density:

∇zlog⁡pσ(z)=E[x∣z]−zσ2=D(z)−zσ2\nabla_{\mathbf{z}}\log p_\sigma(\mathbf{z}) = \frac{\mathbb{E}[\mathbf{x}\mid\mathbf{z}] - \mathbf{z}}{\sigma^2} = \frac{D(\mathbf{z}) - \mathbf{z}}{\sigma^2}
(15)

The right side is the vector from the noisy point to the best guess of the clean point, divided by σ2\sigma^2. The derivation is three lines. pσp_\sigma is pp convolved with a Gaussian; differentiating that Gaussian with respect to z\mathbf{z} brings down a factor (x−z)/σ2(\mathbf{x}-\mathbf{z})/\sigma^2 inside the integral; dividing by pσ(z)p_\sigma(\mathbf{z}) turns the integral into an average over the posterior, which is (E[x∣z]−z)/σ2(\mathbb{E}[\mathbf{x}\mid\mathbf{z}] - \mathbf{z})/\sigma^2. Only the noise has to be Gaussian; the data distribution can be anything.

A one-dimensional check: clean values x=±1x=\pm 1 with equal probability, σ=0.5\sigma = 0.5, and an observed z=0.3z = 0.3. The posterior weights are proportional to e−(0.3−1)2/0.5=0.375e^{-(0.3-1)^2/0.5} = 0.375 and e−(0.3+1)2/0.5=0.034e^{-(0.3+1)^2/0.5} = 0.034, so p(x=1∣z)=0.917p(x{=}1\mid z) = 0.917 and E[x∣z]=0.834\mathbb{E}[x\mid z] = 0.834. The formula gives a score of (0.834−0.3)/0.25=2.135(0.834 - 0.3)/0.25 = 2.135, and differentiating log⁡pσ\log p_\sigma numerically at 0.3 gives 2.135 as well.

Figure 7 makes the same check in two dimensions. It computes the denoiser direction (D−z)/σ2(D-\mathbf{z})/\sigma^2 from the exact posterior mean and the score ∇log⁡pσ\nabla\log p_\sigma from the blurred mixture density, two separate calculations, at a probe you drag.

Figure 7 · the denoiser is the score
σ = 0.80
A Gaussian mixture blurred by σ\sigma (amber heat is pσp_\sigma). Drag the probe. The wide amber arrow is (D(z)−z)/σ2(D(\mathbf{z})-\mathbf{z})/\sigma^2; the thin teal arrow is the score. The cosine readout stays at 1.000 for every probe position and every noise σ.

So the optimal denoiser at noise level σ\sigma and the score of pσp_\sigma are the same function up to subtracting z\mathbf{z} and dividing by σ2\sigma^2. Training either one trains both, and a network trained with DDPM's loss can drive a Langevin sampler, or the reverse SDE of the next section.

The SDE view: forward and reverse in continuous time

DDPM noises in 1000 discrete steps and SMLD over a ladder of noise levels. As the step size goes to zero, both become a stochastic differential equation, which describes a particle moved by a deterministic velocity plus random kicks:

dx=f(x,t)⏟drift dt+g(t)⏟diffusion dwd\mathbf{x} = \underbrace{\mathbf{f}(\mathbf{x},t)}_{\text{drift}}\,dt + \underbrace{g(t)}_{\text{diffusion}}\,d\mathbf{w}
(16)

The drift f\mathbf{f} is the deterministic part. dwd\mathbf{w} is the increment of a Wiener process, the continuous limit of adding independent Gaussian steps (over a time step Δt\Delta t it is Gaussian with variance Δt\Delta t), and the diffusion coefficient g(t)g(t) scales it. In discrete form, one step of (16) adds f Δt\mathbf{f}\,\Delta t and gΔt zg\sqrt{\Delta t}\,\mathbf{z} to x\mathbf{x}.

Running the noising backward is the generative direction. Anderson (1982) showed that the time reversal of (16) is another SDE, with the score of the time-tt marginal ptp_t in its drift:

dx=[ f(x,t)−g(t)2 ∇xlog⁡pt(x) ] dt+g(t) dwˉd\mathbf{x} = \big[\,\mathbf{f}(\mathbf{x},t) - g(t)^2\,\nabla_{\mathbf{x}}\log p_t(\mathbf{x})\,\big]\,dt + g(t)\,d\bar{\mathbf{w}}dx=[ f(x,t)−g(t)2 ∇xlog⁡pt(x) ] dt+g(t) dwˉ\begin{aligned} d\mathbf{x} &= \big[\,\mathbf{f}(\mathbf{x},t) - g(t)^2\,\nabla_{\mathbf{x}}\log p_t(\mathbf{x})\,\big]\,dt \\ &\quad + g(t)\,d\bar{\mathbf{w}} \end{aligned}
(17)

Here dtdt is negative (time runs from TT down to 0) and dwˉd\bar{\mathbf{w}} is a Wiener process running backward. To generate, draw from the noise distribution at time TT and integrate (17) down to 0. The only unknown is the score ∇xlog⁡pt\nabla_{\mathbf{x}}\log p_t, which by (15) is what a trained denoiser at that noise level provides. The score in (17) carries the full g(t)2g(t)^2. Song et al. (2021) also derived a deterministic process with the same marginals ptp_t, the probability-flow ODE, whose score term has 12g(t)2\tfrac{1}{2}g(t)^2 and no noise. The tutorial does not cover the ODE; it samples with the stochastic reverse SDE.5

Figure 8 integrates both on a one-dimensional mixture whose ptp_t is known exactly, from the same 3000 starting draws. With the correct coefficients both histograms track ptp_t all the way down. Swap them, so the SDE gets 12g2\tfrac{1}{2}g^2 and the ODE gets g2g^2, and both end at the wrong distribution. The noise term spreads the density by an amount set by g2g^2; the SDE's drift has to cancel that spreading and also contract the density as much as the forward process expanded it, two halves that add to the full g2g^2.

Figure 8 · reverse SDE and probability-flow ODE
t = 0.40
Paths from the prior (t=1t = 1, left) to the data (t=0t = 0, right) under the VP SDE, with a histogram of 3000 particles at the current time against the exact ptp_t. With the correct coefficients the SDE paths are jagged, the ODE paths smooth, and both histograms track ptp_t. Swapped, the SDE with 12g2\tfrac{1}{2}g^2 ends too wide and the ODE with g2g^2 collapses toward a point.

How the SDE unifies DDPM and SMLD: variance preserving and exploding

DDPM and SMLD are (16) with different f\mathbf{f} and gg. The names below come from Song, Sohl-Dickstein, Kingma, Kumar, Ermon and Poole (2021), the score-based SDE paper.

At matched noise levels the two marginals differ only by a scale factor. A VE sample at level σ\sigma is x0+σϵ\mathbf{x}_0 + \sigma\boldsymbol{\epsilon}. Dividing it by 1+σ2\sqrt{1+\sigma^2} gives

x0+σϵ1+σ2=αˉ x0+1−αˉ ϵ,αˉ=11+σ2\frac{\mathbf{x}_0 + \sigma\boldsymbol{\epsilon}}{\sqrt{1+\sigma^2}} = \sqrt{\bar\alpha}\,\mathbf{x}_0 + \sqrt{1-\bar\alpha}\,\boldsymbol{\epsilon}, \qquad \bar\alpha = \frac{1}{1+\sigma^2}x0+σϵ1+σ2=αˉ x0+1−αˉ ϵ,αˉ=11+σ2\begin{gathered} \frac{\mathbf{x}_0 + \sigma\boldsymbol{\epsilon}}{\sqrt{1+\sigma^2}} = \sqrt{\bar\alpha}\,\mathbf{x}_0 + \sqrt{1-\bar\alpha}\,\boldsymbol{\epsilon}, \\ \bar\alpha = \frac{1}{1+\sigma^2} \end{gathered}

which is the VP sample (5) at the same signal-to-noise ratio αˉ/(1−αˉ)=1/σ2\bar\alpha/(1-\bar\alpha) = 1/\sigma^2. VE level σ=1\sigma = 1 matches αˉ=0.5\bar\alpha = 0.5, DDPM's step 259; DDPM's last step, αˉ1000=4.04×10−5\bar\alpha_{1000} = 4.04\times10^{-5}, matches σ≈157\sigma \approx 157. The tutorial cites Kawar et al. for the observation that VP and VE inference are equivalent, and adds that the hyperparameters still change what is trained. Figure 9 shows both at matched levels.

Figure 9 · VP and VE at the same noise level
ᾱ = 0.500
A two-bump dataset (clean positions as amber ticks) noised to the same signal-to-noise ratio. VP shrinks the bumps toward the center with total variance near 1; VE leaves them in place and widens and flattens as noise σ grows to 157. The dashed curve in the lower panel is the VP curve. Switch to VE ÷ √(1+σ²) and the VE curve lands on it at every σ.

Predictor-corrector sampling

DDPM's update (11) is one discretized step of the reverse VP SDE, and any finite step leaves a numerical error. Song et al.'s predictor-corrector sampler, which the tutorial presents in §4.4, alternates two moves at each noise level. The predictor takes one step of the reverse SDE, which the tutorial calls "an Euler method for the reverse diffusion". The corrector then runs MM Langevin steps (13) at the current level,

x←x+ϵ sθ(x,t)+2ϵ z\mathbf{x} \leftarrow \mathbf{x} + \epsilon\,\mathbf{s}_{\boldsymbol{\theta}}(\mathbf{x}, t) + \sqrt{2\epsilon}\,\mathbf{z}

which pull the samples toward ptp_t at that level before the next predictor step. The tutorial describes the combination as "a hybrid SDE solver coupled with a Markov Chain Monte Carlo". Each corrector step costs one more network evaluation. Figure 10 starts from a predictor output packed too tightly around each mode of a one-dimensional ptp_t. Raise M, then try the smallest and largest step ϵ\epsilon.

Figure 10 · the Langevin corrector
M = 0
ε = 0.015
Samples (4000) against the exact ptp_t at one noise level. At M=0M = 0 they sit too close to the modes (gap 0.41). At ϵ=0.015\epsilon = 0.015, ten corrector steps bring the gap to about 0.05. At ϵ=0.0005\epsilon = 0.0005 sixty steps are not enough. At ϵ=0.12\epsilon = 0.12 the narrow mode empties into the wide one.

A finite step ϵ\epsilon has its own bias: for a Gaussian mode of variance s2s^2, the corrector settles at variance s2/(1−ϵ/(2s2))s^2/(1-\epsilon/(2s^2)), larger than s2s^2, and diverges once ϵ≥2s2\epsilon \ge 2s^2. Small steps are accurate but slow to move the samples; large steps move them fast to the wrong spread.

Four views of one model

Each view describes the same trained network and the same sampler:

Later work builds on these four pieces. Latent diffusion runs the same loss on an autoencoder's latents instead of pixels, classifier-free guidance combines two scores at sampling time, and flow matching regresses a velocity field along a chosen path from noise to data. The current version of the tutorial also has a fifth chapter, on the Langevin and Fokker-Planck equations, which this page does not cover.

Provenance Verified against primary literatureHow we verify
Kingma & Welling (2014)The VAE, the ELBO and the reparameterization trick (tutorial ref [23]). The high-variance verdict on the score-function estimator is theirs (Sec. 2.2), not the tutorial's.
Ho, Jain & Abbeel (2020)DDPM: the forward marginal (5), the posterior (7)-(8), the noise-prediction loss (10), the linear schedule 1e-4 to 0.02 over T = 1000, and the report that x0-prediction gave worse samples early on (ref [16]).
Song & Ermon (2019)NCSN: one network conditioned on sigma, trained over a ladder of noise levels with lambda(sigma) = sigma^2, sampled with annealed Langevin dynamics (ref [40]).
Vincent (2011)Denoising score matching: the kernel score -(x - x')/sigma^2 = -z/sigma as a regression target (ref [43]).
Anderson (1982); Song et al. (2021)The reverse-time SDE (17) with the full g^2; the SDE view with its VP and VE names, the probability-flow ODE (not covered by the tutorial), and the predictor-corrector sampler (refs [2], [42]). The tutorial credits VP/VE to "Song and Ermon [42]", but [42] is the six-author 2021 paper.
Kawar et al. (2022)The observation that VP and VE inference are equivalent (tutorial §4.3, ref [21]); the rescaling by sqrt(1 + sigma^2) shown in Figure 9 is the algebra behind it.
Robbins (1956), Miyasawa (1961), Efron (2011)Tweedie's formula (15). The tutorial uses it without naming it: v1 §2.9 states the denoiser-to-score relation as an approximation proposed by Vincent, and v3 says "predicting noise is equivalent to denoising".
correctionTwo coefficient typos in the tutorial, both taught corrected here. (1) Denoising score matching: v1 prints the target as z/σ² (the derivation line, Eqs 85-86, and 'the noise z/σ²'). With x = x' + σz and z unit-normal, the kernel score −(x−x')/σ² equals −z/σ, so the loss is ‖s_θ(x'+σz) + z/σ‖². The author corrected this in v2 (Eq 106) and v3 (Eq 3.11), which print z/σ. v1 also drops the log in ∇ₓ log p(x) (Eq 79) and ∇ₓ log q(x|x') (Eq 84); one such slip survives in v3's proof of Theorem 3.4. (2) The posterior variance: v3 Eq 2.26 (v1 Eq 37) prints σ_q²(t) = (1−α_t)(1−√ᾱ_{t−1})/(1−ᾱ_t), with a stray square root. The tutorial's own derivation (Eq 2.32, v1 Eq 49) and Ho et al.'s β̃_t have (1−α_t)(1−ᾱ_{t−1})/(1−ᾱ_t). The printed form halves the variance at small t.

Questions you might still have

?

Adding noise destroys the image. How does that help generate one?
The forward noising is used only to make training pairs. You take a clean image, add noise of a known size, and train the network to predict that noise. Generation never runs the forward process: it starts from pure Gaussian noise and applies the learned reverse step about a thousand times.

?

Are DDPM and score-based models the same thing?
Yes, up to a rescaling of the network output. The score of the noised data at level t is minus the noise divided by the square root of 1 minus alpha-bar, so a noise predictor gives a score estimate and the reverse. In continuous time DDPM is the variance-preserving SDE and SMLD the variance-exploding SDE, and dividing a variance-exploding sample by the square root of 1 plus sigma squared turns it into the variance-preserving sample at the same signal-to-noise ratio.

?

Why predict the noise instead of the clean image?
Given the noisy image, the noise and the clean image determine each other through the forward equation, so the two targets carry the same information. Chan motivates the noise target by analogy with residual denoisers, which predict the noise rather than the signal. Ho and coauthors, who introduced DDPM, report that predicting the clean image led to worse sample quality early in their experiments.

?

Does the tutorial cover deterministic sampling?
It covers DDIM (section 2.6 of the current version), a non-Markovian sampler with a free noise scale; setting that scale to zero makes every step deterministic. It does not cover the probability-flow ODE, the deterministic process with the same marginals as the reverse SDE, which comes from Song et al. (2021) and has one half of g squared in front of the score where the reverse SDE has the full g squared. The DDIM and score-based SDE explainers cover both.

?

How many network evaluations does sampling take?
One per reverse step. DDPM with the standard schedule takes 1000 steps, so 1000 forward passes of the network per image. The tutorial quotes Song et al.: more than 1000 hours to generate 50,000 images of size 256 by 256 on a standard GPU. DDIM and other fast samplers cut the step count.

?

Is this the same as flow matching?
They are close relatives. Flow matching regresses a velocity field that carries noise to data along a chosen path, and the deterministic diffusion ODE is one such field. The tutorial stays within the noising and denoising framing; the flow matching explainer covers the other route.

Footnotes & further reading

  1. Variance preservation is a fixed point, not an identity for every input: the total variance stays at exactly 1 only when the data already has unit variance, and otherwise relaxes toward 1 over the chain, which is one reason pixels are rescaled to a fixed range first. This page follows the tutorial's and Ho et al.'s convention, αt\alpha_t per step and αˉt=∏i=1tαi\bar{\alpha}_t = \prod_{i=1}^t \alpha_i cumulative, as on the DDPM page. ↩
  2. The tutorial defines the Stein score ∇xlog⁡pθ(x)\nabla_{\mathbf{x}}\log p_{\boldsymbol{\theta}}(\mathbf{x}) (v3 §3.2, Eq 3.3; v1 Eq 76) separately from the ordinary score ∇θlog⁡pθ(x)\nabla_{\boldsymbol{\theta}}\log p_{\boldsymbol{\theta}}(\mathbf{x}) (Eq 3.4; v1 Eq 77), and notes that maximum likelihood uses the second and Langevin dynamics the first. ↩
  3. Pascal Vincent, A Connection Between Score Matching and Denoising Autoencoders (Neural Computation 23(7), 2011). For the kernel N(x;x′,σ2I)\mathcal{N}(\mathbf{x};\mathbf{x}',\sigma^2\mathbf{I}) the target is (x′−x)/σ2=−z/σ(\mathbf{x}'-\mathbf{x})/\sigma^2 = -\mathbf{z}/\sigma, the target the tutorial prints in v3's Eq 3.11 (v1 printed z/σ2\mathbf{z}/\sigma^2; see the flag above). Song and Ermon's NCSN is explained on the NCSN page. ↩
  4. Tweedie's formula is usually credited to Robbins (1956), who attributes it to Maurice Tweedie, with the Gaussian case also in Miyasawa (1961); see Bradley Efron, Tweedie's Formula and Selection Bias (2011). The word "Tweedie" appears in no version of the tutorial. ↩
  5. Song, Sohl-Dickstein, Kingma, Kumar, Ermon and Poole, Score-Based Generative Modeling through Stochastic Differential Equations (ICLR 2021), explained on the score-based SDE page. The reverse-time SDE is B. Anderson, Reverse-time diffusion equation models (Stochastic Processes and their Applications, 1982). v1 of the tutorial cites the VE probability-flow ODE in passing in §2.9; v3 does not cover it. ↩
  6. The paper: Stanley H. Chan, Tutorial on Diffusion Models for Imaging and Vision (Purdue University; v1 March 2024, v3 January 2025). The VAE is Kingma & Welling, Auto-Encoding Variational Bayes (2014); DDPM is Ho, Jain & Abbeel, Denoising Diffusion Probabilistic Models (2020); DDIM is on the DDIM page; flow matching is on the flow-matching page.