Tutorial on Diffusion Models for Imaging and Vision
Four derivations of the same diffusion model: as a variational autoencoder, as a chain of denoising steps, by score matching, and as a stochastic differential equation.
Stanley Chan's tutorial builds a variational autoencoder, a noise-predicting denoiser, a score model and a stochastic differential equation from first principles, and shows that all four train the same network and sample with the same reverse process.
Explaining the paperTutorial on Diffusion Models for Imaging and VisionA standard diffusion model trains one network to look at an image with a known amount of Gaussian noise added and predict that noise. This page follows the tutorial's four derivations of that loss and of the sampler built on it, with ten figures you can drive.
A generative model for images has one job: produce new images that look like samples from the training set without copying any of them. In probability terms, the training images are draws from an unknown distribution over all possible images, and the model must draw fresh samples from it. A 64×64 color image is a vector of 12,288 numbers, and you hold only a finite set of examples, never the density itself.
Diffusion models work from the opposite direction. Destroying an image is easy: add a little Gaussian noise, then a little more, and after about a thousand steps any image is indistinguishable from pure noise. That direction needs no learning. A network is trained to undo one small step of it, and generation starts from pure noise and applies the learned undo-step a thousand times.
Chan's tutorial (Purdue, 2024; three arXiv versions, the latest January 2025) derives this machine four times, in dependency order:
- the variational autoencoder (VAE), which supplies the latent variable, the evidence lower bound (ELBO) and the reparameterization trick;
- DDPM, which stacks a thousand VAE-like steps and reduces the ELBO to "predict the noise";
- score matching with Langevin dynamics (SMLD), which reaches the same regression target without any noising chain;
- the stochastic differential equation (SDE), in which DDPM and SMLD are two settings of one continuous-time process.
Section numbers and equation numbers below refer to the current version (v3) unless marked v1. The page renumbers the equations it uses as (1) to (17).
How a VAE is trained: the ELBO
A VAE has two networks. The encoder maps an image to a short latent code ; the decoder maps a code back to an image. Training also pushes the codes toward a fixed distribution that is easy to sample, the standard Gaussian . Generation then takes two lines: draw from the Gaussian and decode it.
Both networks output distributions. The encoder is a Gaussian whose mean and per-coordinate variance come from a network with weights . The decoder is a Gaussian centered on the network's reconstruction. A diagonal Gaussian is the default because its parameters are two vectors a network can output, it is sampled in one line, and its KL divergence to the standard Gaussian has a closed form.
The natural objective is the log-likelihood of real images, but it contains an integral over every possible code and cannot be computed. The tutorial splits it into a computable term and a remainder:
The second term is a Kullback-Leibler (KL) divergence, a measure of how far one distribution is from another, and it is never negative. So the first term, the evidence lower bound, can never exceed . The KL in (1) is the gap between the bound and the truth, and it compares the encoder with the true posterior , a distribution nobody can compute. When the encoder equals that posterior, the gap is zero and the bound is exact.
Figure 1 shows the gap for a one-dimensional latent. The slider moves the encoder's Gaussian onto the true posterior; the bar below splits into the ELBO and the KL gap.
Since does not depend on , raising the ELBO by moving the encoder is the same as shrinking the gap. Rearranged, the ELBO splits into the two jobs the VAE does:
The reconstruction term rewards the decoder for rebuilding from its code; with a Gaussian decoder it is a scaled squared error. The prior-matching term penalizes codes that stray from , which keeps the latent space samplable. This KL is against the prior , a different term from the gap in (1), which is against the posterior. The two terms of (2) pull in opposite directions: maximizing the reconstruction term alone favors a distinct, precise code per image, and minimizing the prior KL alone pushes every image's code distribution toward the same Gaussian.
Training needs the gradient of the ELBO with respect to , but enters through a sampling step, and a random draw has no derivative with respect to the parameters of its distribution. The reparameterization trick (Kingma and Welling) moves the randomness into a parameter-free input:
Now carries the randomness, and appears only in a multiply and an add, which backpropagation handles. The alternative, the score-function estimator, differentiates the sampling distribution directly; Kingma and Welling found its gradient "exhibits very high variance" and called it impractical for their purposes. The form of (3), a clean value scaled plus standard noise scaled, is the form the diffusion forward process takes in the next section.
The forward noising process
DDPM (denoising diffusion probabilistic models, Ho, Jain and Abbeel, 2020) changes the VAE in two ways. The latent is the same size as the image: it is the image with noise added. And there is a chain of latents, to , each noisier than the last, with the clean image. The "encoder" has no weights; it adds noise on a fixed schedule:
The per-step scale is a number just below 1. Ho's schedule writes and raises it linearly from at to 0.02 at . The signal is shrunk by and the added noise has variance , and these two are tied: if has unit variance, then has variance . The tutorial picks so that the variance "will not explode and vanish after many iterations".1 A step with signal scale and noise scale repeated forever settles at variance , and makes that 1.
Reaching does not require 400 sequential steps. Each step is a linear map plus independent Gaussian noise, and a sum of independent Gaussians is Gaussian with the variances added. Composing steps therefore gives a scaled copy of plus one Gaussian:
with . The cumulative product says how far along the trip you are. On Ho's schedule, crosses 0.5 near , where signal and noise have equal weight (0.707 each). At , and
At , : what is left is a standard Gaussian with a trace of the image. Equation (5) has the same shape as the reparameterization (3), with in place of the encoder mean and in place of its standard deviation.
How DDPM is trained: from the ELBO to predicting the noise
The model is the reverse direction: a network that takes and produces the distribution of the slightly cleaner . Each reverse step is a Gaussian whose mean a network predicts, a small decoder. The chain is trained by the ELBO, as the VAE was. With the chain in place of a single latent, the tutorial's Theorem 2.4 writes it as three kinds of term:
The first term is reconstruction at the last step, the second is prior matching at (no trainable weights, since the forward process is fixed), and the sum is what the tutorial calls consistency: at every level, the model's reverse step should agree with the true reverse step. The true reverse step is conditioned on the clean image, and with known it is a Gaussian with a closed form (Theorem 2.5). Its mean is a linear combination of where the sample is and where it started:
and its variance is
This is Ho's . The tutorial's Eq 2.26 prints a square root on in the numerator; its own derivation (Eq 2.32) and Ho's paper have none, and (8) follows them. The printed form would give half the correct variance at small (ratio 0.500 at , 0.69 at ).
The model's reverse step uses the same variance , so only the means can differ. The KL between two Gaussians with equal variance is the squared distance between their means divided by , and each consistency term becomes a regression of onto . The target depends on , and (5) solves for in terms of and the noise:
Substituting into (7) turns the target into a function of and , and the tutorial chooses the network's mean to have the same form, with a network in place of the true noise:
The per-step scale sits outside and the cumulative sits under the noise estimate. The difference of the two means is now a multiple of , and the consistency term at level becomes
The training procedure the tutorial writes out drops and samples uniformly, which gives the DDPM loss:
With the weights dropped, (10) is no longer a bound on the log-likelihood; it is a reweighted sum of the same terms, and the reweighting changes which noise levels the network practices on. With from (8), simplifies to : 0.600 at , 0.0103 at , a minimum of 0.00498 at , and 0.0102 at , a factor of 59 from end to end. Under the bound, steps 2 to 100 carry 39.2% of the total weight; under (10) they carry 9.9%, their share of the steps. At the noise standard deviation is 0.015 on a pixel scale of [−1, 1], about two of the 256 intensity levels. Ho et al. explain the choice this way: the small- terms train the network to remove very small amounts of noise, and down-weighting them lets it focus on the harder denoising at larger . They report better sample quality with the unweighted loss. Figure 3 plots the cumulative weight; pick the σₜ² = β̃ₜ tab for the tutorial's variance (8).
Given , the clean image, the noise, and the posterior mean (7) determine one another through (5), so a network that predicts any one of them can be converted to the other two. The tutorial motivates the noise target by analogy with residual denoisers in image processing, which predict the noise rather than the signal. Ho and coauthors also tried predicting and found it led to worse sample quality early in their experiments.
A concrete step on 64×64 color images with Ho's schedule: draw a batch of clean images, shape [B, 3, 64, 64]; draw a level per image, say ; draw of the same shape; scale and add them as in (5), with coefficients 0.442 and 0.897; run the network on to get of shape [B, 3, 64, 64]; take the mean squared error against . The gradient flows into the network's weights only; there is no chain to backpropagate through, because (5) jumps straight to level 400. One network, given as an input, serves all 1000 levels.
# one DDPM training step (Ho's linear schedule, T = 1000)
x0 = sample_batch() # clean images, shape [B, 3, 64, 64]
t = randint(1, T + 1, size=B) # one noise level per image, e.g. 400
eps = randn_like(x0) # the target
xt = sqrt(abar[t]) * x0 + sqrt(1 - abar[t]) * eps # t=400: 0.442, 0.897
loss = ((eps_hat(xt, t) - eps) ** 2).mean() # eq (10), weights dropped
loss.backward() # gradient reaches eps_hat's weights only
opt.step()Sampling from DDPM
Generation runs the chain backward. Start from ; at each level, compute the mean (9) with the trained network and add fresh Gaussian noise of standard deviation :
At on Ho's schedule this reads
The step subtracts 0.00897 times the predicted noise, scales up by 1.0040 to undo the forward shrink of step 400, and adds fresh noise with standard deviation 0.0896. At , because , so the last step adds no noise and returns the mean. The reverse process is stochastic at every other step; two runs from the same end at different images.
# DDPM sampling, eq (11): T network calls per image
x = randn(3, 64, 64) # x_T, pure Gaussian noise
for t in range(T, 0, -1):
e = eps_hat(x, t)
mean = (x - (1 - a[t]) / sqrt(1 - abar[t]) * e) / sqrt(a[t])
z = randn_like(x) if t > 1 else 0
x = mean + sigma_q[t] * z # sigma_q[t]^2 from eq (8)
return x # t=400: 1.0040*(x - 0.00897 e) + 0.0896 zFigure 4 runs (11) on six points in the plane, arranged as two groups of three. In place of a trained network it uses the exact optimal noise predictor for this dataset, so any failure comes from the sampler alone. Play it with 1000 steps and every particle ends on a data point. Then try 10, 2 and 1 steps, and the no noise tab, which drops the term.
With no injected noise, every particle follows the mean, and the means pull toward the average of the data points each particle could still become. The noise keeps particles spread so that they reach the outer points too. With one step, the sampler can only output the posterior mean of given pure noise, which is the center of the data.
The cost is the step count: 1000 network calls per image. The tutorial quotes Song et al.: a DDPM takes more than 1000 hours to generate 50,000 images of size 256×256 on a standard GPU. Section 2.6 of the tutorial (absent from v1) covers DDIM, which replaces the Markov chain with a non-Markovian one that skips levels and has a free noise scale in its update (Eq 2.59). That update's only random term is , so makes sampling deterministic. DDIM reuses a network trained with (10) unchanged. In that section the tutorial switches to DDIM's notation, where means the cumulative product; this page keeps for it throughout.
The score and Langevin dynamics
The tutorial's third chapter starts again from a different question. Given a density , how do you draw samples from it? Score-matching Langevin dynamics (SMLD), from Song and Ermon's NCSN paper (2019), answers with two pieces, and neither mentions a noising chain.
The first piece is the score of the density:
At every point it is a vector pointing in the direction in which the log-density rises fastest. For a one-dimensional Gaussian , the score is : at it is , pointing back toward the mean and longer the farther out you are. The gradient is with respect to the data point, which the tutorial calls the Stein score, as distinct from the score used in maximum-likelihood estimation.2
The score does not depend on the normalizing constant. Any density can be written , where is the integral that makes it sum to one over all 12,288-dimensional images and cannot be computed. Taking the log turns the division into a subtraction of , a constant, and its gradient with respect to is zero, so , and a model of the score never has to estimate .
Figure 5 draws the score of a mixture of six Gaussian clusters after blurring by noise of level . At small each arrow points to its nearest cluster; at large the clusters merge into one broad hill and the arrows point toward its center.
The second piece turns the score into a sampler. Langevin dynamics takes a small step along the score and adds Gaussian noise (the tutorial's Definition 3.1):
The sign is a plus because the step climbs the log-density. Without the noise term, (13) is gradient ascent: every particle climbs to the nearest peak and stops, which finds modes, not samples. With the noise, particles keep moving around the high-density regions, and for a small step size and many iterations the distribution of a particle approaches . The tutorial summarizes it as "gradient descent + noise". The is not a free choice: Figure 6 scales the noise term by a multiplier , and only near does the cloud match the data.
Denoising score matching
Langevin needs the score of the data distribution, and nobody has it: there are training images and no density. A regression needs targets, and the true score cannot be evaluated at any point. Denoising score matching (Vincent, 2011) supplies a target by adding noise of known size. Perturb a clean image to with . The score of that single Gaussian kernel is known exactly:
The belongs to the raw difference , and the to the unit noise ; they are the same quantity in two variables. With and , the offset is , and both forms give the target . A network is trained to output it:
A single target points at one particular clean image and is noisy. Many clean images could have produced the same , and the least-squares fit averages their targets. Vincent proved that this average is the score of the noised distribution , so minimizing (14) trains toward without ever evaluating it.3 The tutorial puts it as "the score function is supposed to take this noisy image and predict the noise ".
Compare (14) with DDPM's (10). Both add Gaussian noise to a clean sample and regress on that noise; (14) asks for where (10) asks for . In DDPM's variables the noise on has standard deviation , and the kernel score is . A trained noise predictor therefore gives a score estimate directly:
One noise level is not enough for sampling. A score fit at a small is accurate only near the data, and Langevin started from noise spends its early steps far from the data, where training samples at that never land. Song and Ermon's noise-conditional score network (NCSN) gives the network as an input and trains it on a ladder of levels , with level 's loss (14) weighted by , chosen as "based on empirical findings". That weight turns (14) into DDPM's loss: multiplying (14) by gives
which is (10)'s squared error, halved, with the noise prediction . Sampling, called annealed Langevin, runs (13) at , then at , and so on down to the smallest level.
Why a denoiser computes the score: Tweedie's formula
DDPM trains a denoiser and SMLD trains a score model, and the previous section showed their losses coincide. A classical identity explains why the trained networks coincide too: Tweedie's formula, which the tutorial uses without naming.4
Take a clean signal from any distribution, add Gaussian noise to get , and ask for the estimate of that minimizes the expected squared error . The minimizer is the posterior mean , the average of all clean signals weighted by how likely each is to have produced , because for any random variable the constant closest to it in mean squared error is its mean. (Version 1 of the tutorial, §2.9, skips this step as "some magical step" and points to a probability textbook; that section was removed in v2.) Tweedie's formula relates this denoiser to the score of the noised density:
The right side is the vector from the noisy point to the best guess of the clean point, divided by . The derivation is three lines. is convolved with a Gaussian; differentiating that Gaussian with respect to brings down a factor inside the integral; dividing by turns the integral into an average over the posterior, which is . Only the noise has to be Gaussian; the data distribution can be anything.
A one-dimensional check: clean values with equal probability, , and an observed . The posterior weights are proportional to and , so and . The formula gives a score of , and differentiating numerically at 0.3 gives 2.135 as well.
Figure 7 makes the same check in two dimensions. It computes the denoiser direction from the exact posterior mean and the score from the blurred mixture density, two separate calculations, at a probe you drag.
So the optimal denoiser at noise level and the score of are the same function up to subtracting and dividing by . Training either one trains both, and a network trained with DDPM's loss can drive a Langevin sampler, or the reverse SDE of the next section.
The SDE view: forward and reverse in continuous time
DDPM noises in 1000 discrete steps and SMLD over a ladder of noise levels. As the step size goes to zero, both become a stochastic differential equation, which describes a particle moved by a deterministic velocity plus random kicks:
The drift is the deterministic part. is the increment of a Wiener process, the continuous limit of adding independent Gaussian steps (over a time step it is Gaussian with variance ), and the diffusion coefficient scales it. In discrete form, one step of (16) adds and to .
Running the noising backward is the generative direction. Anderson (1982) showed that the time reversal of (16) is another SDE, with the score of the time- marginal in its drift:
Here is negative (time runs from down to 0) and is a Wiener process running backward. To generate, draw from the noise distribution at time and integrate (17) down to 0. The only unknown is the score , which by (15) is what a trained denoiser at that noise level provides. The score in (17) carries the full . Song et al. (2021) also derived a deterministic process with the same marginals , the probability-flow ODE, whose score term has and no noise. The tutorial does not cover the ODE; it samples with the stochastic reverse SDE.5
Figure 8 integrates both on a one-dimensional mixture whose is known exactly, from the same 3000 starting draws. With the correct coefficients both histograms track all the way down. Swap them, so the SDE gets and the ODE gets , and both end at the wrong distribution. The noise term spreads the density by an amount set by ; the SDE's drift has to cancel that spreading and also contract the density as much as the forward process expanded it, two halves that add to the full .
How the SDE unifies DDPM and SMLD: variance preserving and exploding
DDPM and SMLD are (16) with different and . The names below come from Song, Sohl-Dickstein, Kingma, Kumar, Ermon and Poole (2021), the score-based SDE paper.
- DDPM is the variance-preserving (VP) SDE: and . The DDPM step scales by and adds noise , and for small , so it moves by plus noise of variance , one Euler step of the SDE with . The drift shrinks the signal at the rate the noise refills the variance, so unit-variance data keeps unit variance.
- SMLD is the variance-exploding (VE) SDE: and . Its discrete chain adds noise to at each level: no shrinking, and the noise variance accumulates to , which grows without bound as the ladder goes up.
At matched noise levels the two marginals differ only by a scale factor. A VE sample at level is . Dividing it by gives
which is the VP sample (5) at the same signal-to-noise ratio . VE level matches , DDPM's step 259; DDPM's last step, , matches . The tutorial cites Kawar et al. for the observation that VP and VE inference are equivalent, and adds that the hyperparameters still change what is trained. Figure 9 shows both at matched levels.
Predictor-corrector sampling
DDPM's update (11) is one discretized step of the reverse VP SDE, and any finite step leaves a numerical error. Song et al.'s predictor-corrector sampler, which the tutorial presents in §4.4, alternates two moves at each noise level. The predictor takes one step of the reverse SDE, which the tutorial calls "an Euler method for the reverse diffusion". The corrector then runs Langevin steps (13) at the current level,
which pull the samples toward at that level before the next predictor step. The tutorial describes the combination as "a hybrid SDE solver coupled with a Markov Chain Monte Carlo". Each corrector step costs one more network evaluation. Figure 10 starts from a predictor output packed too tightly around each mode of a one-dimensional . Raise M, then try the smallest and largest step .
A finite step has its own bias: for a Gaussian mode of variance , the corrector settles at variance , larger than , and diverges once . Small steps are accurate but slow to move the samples; large steps move them fast to the wrong spread.
Four views of one model
Each view describes the same trained network and the same sampler:
- As a VAE, the forward chain is a fixed encoder with 1000 latents, each reverse step is a Gaussian decoder with variance from (8), and training maximizes the ELBO (6).
- As DDPM, each consistency term of the ELBO is a weighted squared error on the noise, and dropping the weights gives the loss (10): add noise at a random level, predict it.
- As score matching, the same regression with weight trains a score model (the noise prediction divided by ), and Tweedie's formula (15) says the optimal denoiser and the score are one function. Langevin dynamics (13) samples with it.
- As an SDE, DDPM is the VP setting and SMLD the VE setting of (16), equal up to division by , and generation integrates the reverse-time SDE (17) with the trained network as its score, with Langevin steps as an optional corrector.
Later work builds on these four pieces. Latent diffusion runs the same loss on an autoencoder's latents instead of pixels, classifier-free guidance combines two scores at sampling time, and flow matching regresses a velocity field along a chosen path from noise to data. The current version of the tutorial also has a fifth chapter, on the Langevin and Fokker-Planck equations, which this page does not cover.
Questions you might still have
Adding noise destroys the image. How does that help generate one?
The forward noising is used only to make training pairs. You take a clean image, add noise of a known size, and train the network to predict that noise. Generation never runs the forward process: it starts from pure Gaussian noise and applies the learned reverse step about a thousand times.
Are DDPM and score-based models the same thing?
Yes, up to a rescaling of the network output. The score of the noised data at level t is minus the noise divided by the square root of 1 minus alpha-bar, so a noise predictor gives a score estimate and the reverse. In continuous time DDPM is the variance-preserving SDE and SMLD the variance-exploding SDE, and dividing a variance-exploding sample by the square root of 1 plus sigma squared turns it into the variance-preserving sample at the same signal-to-noise ratio.
Why predict the noise instead of the clean image?
Given the noisy image, the noise and the clean image determine each other through the forward equation, so the two targets carry the same information. Chan motivates the noise target by analogy with residual denoisers, which predict the noise rather than the signal. Ho and coauthors, who introduced DDPM, report that predicting the clean image led to worse sample quality early in their experiments.
Does the tutorial cover deterministic sampling?
It covers DDIM (section 2.6 of the current version), a non-Markovian sampler with a free noise scale; setting that scale to zero makes every step deterministic. It does not cover the probability-flow ODE, the deterministic process with the same marginals as the reverse SDE, which comes from Song et al. (2021) and has one half of g squared in front of the score where the reverse SDE has the full g squared. The DDIM and score-based SDE explainers cover both.
How many network evaluations does sampling take?
One per reverse step. DDPM with the standard schedule takes 1000 steps, so 1000 forward passes of the network per image. The tutorial quotes Song et al.: more than 1000 hours to generate 50,000 images of size 256 by 256 on a standard GPU. DDIM and other fast samplers cut the step count.
Is this the same as flow matching?
They are close relatives. Flow matching regresses a velocity field that carries noise to data along a chosen path, and the deterministic diffusion ODE is one such field. The tutorial stays within the noising and denoising framing; the flow matching explainer covers the other route.
Footnotes & further reading
- Variance preservation is a fixed point, not an identity for every input: the total variance stays at exactly 1 only when the data already has unit variance, and otherwise relaxes toward 1 over the chain, which is one reason pixels are rescaled to a fixed range first. This page follows the tutorial's and Ho et al.'s convention, per step and cumulative, as on the DDPM page. ↩
- The tutorial defines the Stein score (v3 §3.2, Eq 3.3; v1 Eq 76) separately from the ordinary score (Eq 3.4; v1 Eq 77), and notes that maximum likelihood uses the second and Langevin dynamics the first. ↩
- Pascal Vincent, A Connection Between Score Matching and Denoising Autoencoders (Neural Computation 23(7), 2011). For the kernel the target is , the target the tutorial prints in v3's Eq 3.11 (v1 printed ; see the flag above). Song and Ermon's NCSN is explained on the NCSN page. ↩
- Tweedie's formula is usually credited to Robbins (1956), who attributes it to Maurice Tweedie, with the Gaussian case also in Miyasawa (1961); see Bradley Efron, Tweedie's Formula and Selection Bias (2011). The word "Tweedie" appears in no version of the tutorial. ↩
- Song, Sohl-Dickstein, Kingma, Kumar, Ermon and Poole, Score-Based Generative Modeling through Stochastic Differential Equations (ICLR 2021), explained on the score-based SDE page. The reverse-time SDE is B. Anderson, Reverse-time diffusion equation models (Stochastic Processes and their Applications, 1982). v1 of the tutorial cites the VE probability-flow ODE in passing in §2.9; v3 does not cover it. ↩
- The paper: Stanley H. Chan, Tutorial on Diffusion Models for Imaging and Vision (Purdue University; v1 March 2024, v3 January 2025). The VAE is Kingma & Welling, Auto-Encoding Variational Bayes (2014); DDPM is Ho, Jain & Abbeel, Denoising Diffusion Probabilistic Models (2020); DDIM is on the DDIM page; flow matching is on the flow-matching page.
How could this explainer be improved? Found an error, or something unclear? I read every message.