Deep Unsupervised Learning using Nonequilibrium Thermodynamics

Erase data into noise in a thousand tiny steps, then learn to put it back one step at a time.

Each step adds so little noise that undoing it is a simple, learnable nudge. Chain the learned reversals and a formless Gaussian turns back into data you can sample and score.

An ink drop spreads through a glass of water until the glass is an even gray. That much is easy; it happens on its own. Running the film backward, watching the gray gather itself back into a sharp drop, is the hard part. This paper works out how to learn that backward film, and every diffusion model since starts here.

This is the 2015 paper that started diffusion models. Everything that came later, DDPM, score-based SDEs, Stable Diffusion, the diffusion policies steering robots, traces back to the idea worked out here. It is worth reading in its own terms, because the popular retelling has folded in tricks that arrived years later. What this paper actually proposes is both simpler and stranger than the version now in common circulation.

A few ideas carry it: why flexibility and tractability usually fight, how adding noise slowly turns a hard problem into an easy one, why the reverse of a tiny noising step is itself a simple Gaussian, how you train the reverse steps against a target you can write down in closed form, and why a detour through statistical physics hands you an exact-in-the-limit likelihood. None of them is hard alone. Together they are the first tractable, flexible, deep generative model.

Flexible or tractable, pick one

Start with the problem the paper is trying to solve, because the design answers it. You want to model some complicated distribution of data, say the distribution of natural images, with a family of probability distributions. Historically you could have flexibility or you could have tractability, rarely both.

Tractable means you can actually compute with the model: evaluate the probability it assigns to a data point, fit it to data, and draw samples, all cheaply. A single Gaussian is tractable. So is a Laplace. The trouble is they are rigid; a Gaussian cannot capture the structure of faces or handwriting.

Flexible means the family can bend to fit any structure you throw at it. The usual way to get flexibility is to define a distribution through any non-negative function $\phi(\mathbf{x})$, and normalize it: $p(\mathbf{x}) = \phi(\mathbf{x})/Z$. Now the model can take almost any shape. But that innocent $Z$ in the denominator, the normalizing constant that makes the probabilities sum to one, is an integral of $\phi$ over the entire space, and for anything expressive it is hopeless to compute. Evaluate, train, or sample from such a model and you are usually running an expensive Monte Carlo process that only approximates the answer.

The paper's move is to buy both at once, using a construction from physics. Instead of writing down one flexible-but-unnormalizable distribution, it defines the model as the endpoint of a process: a long chain of small, individually simple, individually tractable steps. Each step is a Gaussian you can evaluate and sample in closed form, so the whole chain is too. The flexibility comes from stacking a thousand of them; the tractability comes from each one being trivial. The rest of the paper is what that idea costs and what it buys.

The forward process: dissolve data into noise

The construction has two halves. A fixed forward process that destroys the data, and a learned reverse process that restores it. The forward half is the easy one, and it requires no learning at all.

Take a data point $\mathbf{x}^{(0)}$ and apply a diffusion step: shrink it a little and add a little Gaussian noise. Do that $T$ times. Each step is the same kind of small perturbation, and after enough of them the data has washed out into a featureless standard Gaussian, the same blob no matter which image you started from. The single forward step is one Gaussian transition:

Read it slowly, because one detail here is load-bearing. The mean of the new sample is the old one scaled down by $\sqrt{1-\beta_t}$, and the noise added has variance $\beta_t$. The signal is not left alone with noise piled on top; it is shrunk by exactly the amount that makes room for the noise. The two moves balance: for data scaled to unit variance, the total variance of $\mathbf{x}^{(t)}$ stays pinned at one all the way along, and the chain glides to a clean $\mathcal{N}(\mathbf{0},\mathbf{I})$ instead of drifting off to infinity. (Later work named this the variance-preserving path, to contrast it with the variance-exploding recipe that leaves the signal untouched and lets the noise grow without bound. Both names are retrospective, from the score-SDE paper in 2021; neither appears here.)

The schedule $\beta_1,\dots,\beta_T$ sets how fast the noise comes in. Keep it small and the destruction is gradual, which, as the next section shows, is exactly what keeps the reverse learnable. Below is the paper's own toy example, a two-dimensional swiss roll. Drag the step from clean data to pure noise and watch the structure dissolve while the total variance holds near one:

A clarification saves confusion later. This paper only ever writes the single small step, equation (1). If you want the noised sample at step $t$ directly from the clean data, you compose $t$ of these Gaussians by hand, and they collapse into one Gaussian with a signal coefficient that is a running product of the per-step factors. That compact one-shot form, the $\sqrt{\bar\alpha_t}$ notation everyone quotes today, is not in this paper; it was introduced by DDPM five years later. The figure above and the code further down use it because it is convenient, but the paper itself thinks one tiny step at a time.

Why reversing it is easy: keep the steps tiny

Generation means running the arrow backward: start from the Gaussian blob and undo the noise, step by step, until you land back on the data. To do that you need the reverse of a diffusion step, $q(\mathbf{x}^{(t-1)} \mid \mathbf{x}^{(t)})$. The entire method rests on a single fact about it.

If the step is small enough, the reverse of a Gaussian diffusion step is itself a Gaussian. This is a classical result the paper attributes to William Feller (1949). Its consequence is enormous: to describe the reverse process, you do not need some wild, general-purpose distribution. You need a Gaussian, and a Gaussian is pinned down by just its mean and covariance. So the entire learning problem shrinks to: at each step, predict the mean and covariance of a Gaussian. That is plain regression.

Why does the step have to be small? Because for a big jump the reverse is not Gaussian. Take a data distribution with two clumps. You are handed a noisy point and asked which clean point it came from. If only a whisker of noise was added, the answer is nearly certain: it came from right about here, a single tight bump, well described by one Gaussian. But if a large blast of noise was added, the noisy point could plausibly have come from either clump, and the honest reverse distribution has two peaks. No single Gaussian can fit a two-peaked target. Drag the step size and watch the true reverse posterior split:

So the design is a trade. Take tiny steps and each reverse step is a clean Gaussian you can learn by regression, but you need a great many of them; the paper uses $T = 1000$ steps for its image models. Take big steps and the chain is short, but each reverse step is a hard multimodal thing a Gaussian cannot represent, and the model breaks. Cheap-per-step and many, over expensive-per-step and few. Diffusion chooses the first, which is why these models are deep, often with thousands of layers or time steps, and why that depth was itself a selling point in 2015 when most generative models were a handful of layers.

The reverse step the network has to produce is therefore

where $\mathbf{f}_\mu$ and $\mathbf{f}_\Sigma$ are the learned mean and covariance functions, one shared network reading the current noisy sample and the step index $t$. This network predicts the reverse Gaussian's mean and covariance directly. It does not predict the noise $\boldsymbol{\epsilon}$. Noise-prediction, and the simpler loss that comes with it, are DDPM's contribution in 2020, not this paper's. Here the target of estimation is the mean and covariance of a sequence of Gaussians, and estimating a probability distribution has been reduced to regression.

Generation: run the chain backward

Suppose the reverse steps are trained. Generation is then a straight walk down the chain. Draw a sample of pure noise from the prior $\pi = \mathcal{N}(\mathbf{0},\mathbf{I})$. Ask the network for the reverse mean and covariance at the top step, sample $\mathbf{x}^{(T-1)}$ from that Gaussian, and repeat, walking the noise level down until you reach $\mathbf{x}^{(0)}$, a sample from the data. Because a fresh draw is taken at every step, the sampler is ancestral: each step's output feeds the next, noise and all.

The best way to see it is to watch the field. Press play and a cloud of pure-noise particles is drawn back onto the swiss roll. The teal arrows are the reverse drift, $\mathbf{f}_\mu(\mathbf{x}, t) - \mathbf{x}$, the paper's own Figure 1 (bottom row): the direction each point is pushed at that noise level. Watch how it reorganizes as the noise falls:

The drift keeps changing direction on the way down for a simple reason. Early on, with heavy noise on the input, the best guess of the clean point is close to the average of all the data, so every particle gets pulled toward one central blob and the field is smooth and slow. As the noise falls, enough signal survives to place a point on a specific arm of the roll, and the field reorganizes from "everything to the center" into "each region to its own arm." Diffusion paints coarse first and fine last, and this is why: the layout is decided at high noise, the detail at low noise.

Training: match each step to a known target

We still owe an answer to the real question: how do you train the reverse means and covariances when the true reverse of the data is exactly the thing you do not know? The paper's answer is a variational bound, the same shape of argument that trains a variational autoencoder.

The probability the model assigns to a data point is an integral over all the intermediate noisy states, $p(\mathbf{x}^{(0)}) = \int d\mathbf{x}^{(1\cdots T)}\, p(\mathbf{x}^{(0\cdots T)})$, which is intractable head on. The trick, borrowed from annealed importance sampling, is to multiply and divide by the forward trajectory $q(\mathbf{x}^{(1\cdots T)} \mid \mathbf{x}^{(0)})$, which you do know, and rewrite the likelihood as an average over forward paths of a ratio of reverse to forward probabilities. Jensen's inequality, the fact that the log of an average is at least the average of the log, then turns that into a lower bound you can actually optimize:

$K$ is a lower bound on the log-likelihood, so training maximizes it (a small subtlety to keep straight: the loss you hand an optimizer is $-K$). And it simplifies into something you can compute term by term. The messy ratio reorganizes into a sum of KL divergences plus a few fixed entropy terms:

Read the pieces slowly. The term the network controls is $p(\mathbf{x}^{(t-1)} \mid \mathbf{x}^{(t)})$, its reverse step. The thing it is compared against is $q(\mathbf{x}^{(t-1)} \mid \mathbf{x}^{(t)}, \mathbf{x}^{(0)})$, the forward process run one step backward while also being told the clean starting point $\mathbf{x}^{(0)}$. That extra conditioning is what makes it tractable: once you know where you started, the one-step reverse is a plain Gaussian you can write down exactly. So every step of the chain has a known Gaussian target, and training is nothing more than pushing each learned reverse Gaussian onto its known target, measured by KL. That trailing constant lumps together a few entropy terms that depend only on the fixed forward process and the fixed prior; they set the ceiling but never move during training.

Notice the payoff of Feller's result cashing out here. Because each target is a single Gaussian (small steps), each KL is between two Gaussians, which has a closed form, so the entire bound is analytic. Drag the slider to improve the reverse kernels and watch every per-step mismatch shrink while the bound climbs toward the true log-likelihood:

To make it concrete, one training step on a real example. Take a 32×32 color image from CIFAR-10, a point $\mathbf{x}^{(0)}$ in $32\times32\times3 = 3072$ dimensions, scaled to unit variance. Pick a step, say $t = 500$ of $1000$. Build the noised input $\mathbf{x}^{(500)}$ by scaling $\mathbf{x}^{(0)}$ down and mixing in fresh Gaussian noise, so it is roughly half signal, half noise. Feed $\mathbf{x}^{(500)}$ and the number $500$ to the network; it returns a 3072-dimensional mean and a (diagonal) 3072-dimensional covariance for $\mathbf{x}^{(499)}$. Compute the KL between that Gaussian and the known forward-posterior Gaussian, a single number, and backpropagate. In code:

# one training step: maximize the bound K over a minibatch
x0     = sample_batch()               # clean data, scaled to variance 1
t      = randint(2, T)                # a step to score
eps    = randn_like(x0)               # frozen noise (held fixed for grads)
xt     = sqrt(abar[t]) * x0 + sqrt(1 - abar[t]) * eps   # abar[t]=prod(1-beta[:t])
mu_q, s_q = forward_posterior(x0, xt, t)   # q(x_{t-1}|x_t,x0): a known Gaussian
mu_p, s_p = net(xt, t)                     # p(x_{t-1}|x_t): learned mean + cov
loss   = kl_gauss(mu_q, s_q, mu_p, s_p)    # one -K term; average the batch
loss.backward(); opt.step()                # descend the KL = ascend the bound K

The noise $\boldsymbol{\epsilon}$ is drawn once and then held fixed while the gradient is taken, a reparameterization the paper calls "frozen noise" and that Kingma and Welling introduced the year before. It lets the gradient flow through the sampling. And sampling is the reverse of training: no forward posterior, no clean target, just the learned chain rolled out from noise.

# sampling: ancestral rollout of the learned reverse chain
x = randn(n, d)                       # start from the prior  pi = N(0, I)
for t in range(T, 0, -1):             # walk noise down to data
    mu, sigma = net(x, t)             # learned reverse mean and covariance
    z = randn(n, d) if t > 1 else 0   # fresh noise each step (ancestral)
    x = mu + sigma * z                # sample x_{t-1} ~ p(x_{t-1}|x_t)
return x                              # a sample from the data distribution

The physics: an exact likelihood from one path

The title says nonequilibrium thermodynamics. The forward process, slowly destroying structure and raising entropy toward a featureless equilibrium, is literally a diffusion in the physicist's sense, and the tools for reasoning about it were built for exactly that. The connection buys two concrete things.

The first is the likelihood estimator itself. That ratio-of-trajectories average in equation (3) is an annealed importance sampling weight (Neal, 2001), the standard way physicists and statisticians estimate an intractable partition function by slowly morphing a simple distribution into a hard one. The Jarzynski equality, a result from nonequilibrium statistical physics, is the same identity in different clothing: it relates the work done along many fast, random paths to a single equilibrium free-energy difference. Here it says the log of the average ratio is the log-likelihood, and Jensen's inequality plays the role of the second law of thermodynamics, that average work is at least the free-energy change.

The second is when the bound is tight. In a quasi-static process, one run so slowly that it stays reversible, the forward and reverse trajectories become the same distribution, the ratio stops fluctuating, and a single sampled path evaluates the likelihood exactly. In the model, that limit is the one where the steps go to zero and the chain gets infinitely long. So the variational bound is not just some slack inequality; it is exact in the idealized reversible limit, and the smaller your steps, the tighter it gets. The same physical picture, take it slowly and you lose nothing, is why the method needs many tiny steps rather than a few big ones, seen now from the likelihood side rather than the Gaussian-reverse side.

A caution on reading the reported numbers. What the paper reports is the bound $K$, not the exact likelihood. On CIFAR-10 that bound is $5.4 \pm 0.2$ bits per pixel. Since $K$ only ever undershoots the true log-likelihood, the real model is at least this good, and in the reversible limit the two coincide.

Multiplying distributions: inpaint and guide

The structure has one more property, and it became the seed of classifier guidance. Suppose you have a trained model $p(\mathbf{x})$ and you want to condition it: fill in a missing region of an image, denoise a corrupted one, or steer generation toward some property. All of these are the same operation, multiplying the model by a second function $r(\mathbf{x})$ and renormalizing:

For most generative models this is a nightmare, because multiplying two distributions and renormalizing means computing a new normalizing constant, exactly the intractable integral you were avoiding. Under a diffusion model it is surprisingly cheap. Because each reverse step is a sharply-peaked Gaussian, the factor $r$ can be folded into every step of the reverse chain (not applied once at the end), and a smooth $r$ only shifts a Gaussian's mean, leaving the covariance alone:

The reverse mean is nudged by the covariance times the gradient of $\log r$, evaluated at the unperturbed mean. If you have seen classifier guidance, that expression should look familiar, because it is the same one: steering a diffusion model by the gradient of a log-conditional was invented here, in 2015, as "multiplying distributions." Dhariwal and Nichol's 2021 classifier guidance is this mechanism plus a tunable strength; classifier-free guidance is its grandchild. The exactness has limits: it is a first-order approximation for a general smooth $r$, and exact only when $r$ multiplies a Gaussian in closed form, which covers the two cases the paper actually uses, a hard constraint for inpainting and a Gaussian for denoising.

The figure makes the inpainting case tangible. The model is the swiss roll; the observation is "the horizontal coordinate is about here." The posterior is the product, and because a spiral crosses a vertical line more than once, it is naturally multimodal, lighting up several compatible arcs at once. Drag the band, and narrow it from a vague hint to a hard constraint:

The paper demonstrates exactly this: inpainting a 100×100 hole in a bark-texture image by pinning the known pixels and sampling the missing ones from the model posterior, with long-range structure like a crack running coherently into the filled region. Conditioning that most models treat as a separate, hard inference problem is, for diffusion, just another reverse chain with a shifted mean.

What it did, and what came after

For 2015 the results were strong for a method this general. On the dead-leaves dataset, a standard stand-in for natural image statistics, the model reached $1.489$ bits per pixel against the previous best of $1.244$, state of the art at the time (log-likelihoods, so higher is better). On MNIST, scored with the same Parzen-window estimate everyone used then, it landed at $317$ bits, in the same range as the contemporary methods including the brand-new adversarial network at $325$; comparable, not a clear win, and on a metric noisy enough that the ordering barely means anything. The headline was never a single benchmark. It was that one method, trained the same way, could fit a swiss roll, a binary sequence, MNIST digits, CIFAR-10, and natural textures, and could sample from and evaluate all of them.

What this paper did not have is as instructive as what it did, because the gap is the history of the field. It learns the reverse mean and covariance directly and trains the full variational bound, term by term. It has no score function, no Langevin sampling, no $\sqrt{\bar\alpha_t}$ shortcut, no simplified squared-error loss. Those arrived later, and each one made the recipe cheaper or sharper. DDPM (2020) reparameterized the reverse step to predict the noise $\boldsymbol{\epsilon}$, fixed the reverse variance instead of learning it, and replaced the bound with a plain denoising loss, which is what finally made the samples crisp. NCSN (2019) came at the same target from the score-matching side, and Score-SDE (2021) unified both under stochastic differential equations. Everything downstream, Stable Diffusion, classifier-free guidance, flow matching, the diffusion policies driving robots, is built on the three-part frame set here: a fixed forward corruption, a learned reverse, and a variational objective that ties them together.

The lasting idea is the one the title points at. A hard modeling problem, learn a flexible distribution you can still sample and score, becomes an easy one if you refuse to solve it in a single leap. Destroy the structure slowly, and each small piece of the reverse is simple enough to learn on its own. Diffusion models are, at bottom, that patience made into an algorithm.