Score SDE: Score-Based Generative Modeling through Stochastic Differential Equations

Gradually noising data is a stochastic differential equation, and running that equation backward with a learned score generates new data.

Score matching with Langevin dynamics and denoising diffusion turn out to be two discretizations of one family of equations. The same learned score then gives a sampler with an error-correcting step, a deterministic ODE with exact likelihoods, and conditional generation from an unconditional model.

In 2019 a score-matching model reached an FID (a distance between generated and real image statistics; lower is better) of 25.32 on CIFAR-10; in 2020 a denoising diffusion model reached 3.17, by what looked like a different method. Treating both as limits of continuous noising processes, the paper gets the FID down to 2.20.

By late 2020 two families of generative models trained on noised images. Score matching with Langevin dynamics (SMLD), the line of NCSN, added Gaussian noise at a ladder of growing scales, trained one network to estimate the gradient of the log-density at every scale, and generated by walking down the ladder with a noisy gradient-following sampler. DDPM defined a 1000-step chain that shrinks an image slightly and adds a little noise at each step, trained a network to undo one step, and generated by running the chain in reverse.

Song and coauthors let the number of noise scales go to infinity. Infinitely many, infinitely close scales make the noising a continuous process driven by random noise, which is a stochastic differential equation (SDE). A 1982 result of Brian Anderson gives the SDE that runs any such process backward in time, and that reverse equation contains exactly one unknown: the gradient of the log-density of the noised data, called the score, at every noise level. SMLD and DDPM were both already training networks to estimate that quantity.

The page follows the paper's order:

• the forward SDE, which turns data into noise and has nothing to learn;
• the reverse SDE, which turns noise into data and needs the score;
• denoising score matching, which learns the score by regression;
• VE, VP and sub-VP, the three forward SDEs the paper uses, two of which are SMLD and DDPM;
• the probability-flow ODE, a deterministic process with the same marginals, and its exact likelihoods;
• predictor-corrector sampling, and conditional generation by adding one gradient.

The forward SDE: noising data as a continuous process

A 32×32 color image is a vector of 3072 numbers, a point in 3072-dimensional space. Real images occupy a tiny region of that space: pick 3072 random numbers and you get static. Generating an image means producing a point inside that region, starting from something easy to sample.

The paper builds the path in the opposite direction first. A process $\mathbf{x}(t)$ runs over time $t \in [0, T]$. At $t = 0$ it is a data sample, drawn from $p_0 = p_{\text{data}}$. Noise is mixed in continuously, and by $t = T$ the distribution $p_T$ is a fixed Gaussian, the prior, which carries no information about the data and is trivial to sample. The rule for one infinitesimal step is an Itô SDE:

$\mathbf{f}(\mathbf{x}, t)$ is the drift, a deterministic velocity that can depend on position. $g(t)$ is the diffusion coefficient, a scalar that sets how much noise enters at time $t$. $\mathbf{w}$ is a standard Wiener process (Brownian motion): over a step of length $\Delta t$, its increment is Gaussian with variance $\Delta t$ in every coordinate. The paper takes $g$ to be a scalar that does not depend on $\mathbf{x}$ for ease of presentation and generalizes to a $d \times d$ matrix $\mathbf{G}(\mathbf{x}, t)$ in its Appendix A. The forward SDE has no trainable parameters. The designer picks $\mathbf{f}$ and $g$ so that $p_T$ ends up close to a known Gaussian.

To simulate (5) on a computer, replace $dt$ with a small step $\Delta t$ and $d\mathbf{w}$ with $\sqrt{\Delta t}\,\mathbf{z}$, $\mathbf{z} \sim \mathcal{N}(0, I)$ (the Euler-Maruyama method). For a worked step, take the forward SDE that DDPM turns into, with $\mathbf{f} = -\tfrac{1}{2}\beta(t)\mathbf{x}$, $g = \sqrt{\beta(t)}$, and the paper's schedule $\beta(t) = 0.1 + 19.9\,t$ on $t \in [0, 1]$. At $t = 0.5$, $\beta = 10.05$, and with $\Delta t = 0.001$ one step is

A pixel at 0.8 moves to 0.796 plus a Gaussian kick of standard deviation 0.10. A thousand such steps from $t = 0$ to $t = 1$ multiply the original signal by $e^{-5.025}$ (half of $\int_0^1 \beta = 10.05$ in the exponent), which is 0.0066, so what remains at $t = 1$ is 0.66% of the image plus noise of variance $1 - e^{-10.05}$, a standard Gaussian to four decimals.

Drag $t$ in Figure 1 to run this SDE on three clusters of 2D points. The clusters spread and merge into one blob at the origin, and the white path is one particle's simulated trajectory, which wanders because every step adds a fresh kick.

At $t = 1$ the cloud no longer depends on which cluster a particle started in. Going back has to restore that information at every noise level, and the next section shows that the one quantity it needs is the score.

How the reverse-time SDE generates data

A random process played backward in time is in general a different kind of process. Anderson (1982) proved that diffusions are an exception: the time-reversal of a diffusion is again a diffusion, with the same noise coefficient and a drift that can be written down. For the forward SDE (5), the reverse-time SDE is

Time runs from $T$ down to $0$, so $dt$ is a negative step, and $\bar{\mathbf{w}}$ is a Wiener process for time flowing backward. $p_t$ is the distribution of the noised data at time $t$, the cloud in Figure 1 at that slider position. The noise term is the same $g(t)$ as in the forward SDE. The drift is the forward drift minus $g(t)^2$ times one new quantity, $\nabla_{\mathbf{x}}\log p_t(\mathbf{x})$. The coefficient is the full $g(t)^2$, with no factor of one half; the half appears later, in the ODE (13).

$\nabla_{\mathbf{x}}\log p_t(\mathbf{x})$ is the score of $p_t$: the gradient of the log-density with respect to the data point $\mathbf{x}$, not with respect to any model parameters. It is a vector field with one arrow per point in data space, and each arrow points in the direction in which the noised density increases fastest. With the score known at every $t$, generation is simulation: draw $\mathbf{x}(T)$ from the prior and integrate (6) down to $t = 0$.

The score does not need the density's normalizing constant. A model density is usually written $\tilde p(\mathbf{x})/Z$, where $Z$ is the integral of $\tilde p$ over all 3072 dimensions and cannot be computed for a neural network. Taking the gradient of the log removes it:

because $Z$ is a constant. A network that outputs a 3072-dimensional vector can represent a score directly, with no constraint on its architecture. Normalizing flows, which model the density itself, have to restrict their layers to invertible maps with cheap Jacobian determinants so that the density stays normalized and computable.

For one concrete reverse step, use the noising SMLD turns into, which has $\mathbf{f} = 0$ and noise levels $\sigma_i$. Discretizing (6) between adjacent levels gives the paper's reverse-diffusion update for this case (Algorithm 2, lines 3 and 5):

Take one-dimensional data distributed as $\mathcal{N}(0, 1)$. Noised to level $\sigma$, it is $\mathcal{N}(0, 1 + \sigma^2)$, whose score is $-x/(1 + \sigma^2)$. The CIFAR-10 schedule spaces 1000 levels geometrically from 0.01 to 50, so adjacent levels differ by a factor of 1.00856. Stepping from $\sigma_{i+1} = 10$ to $\sigma_i = 9.915$ gives $\sigma_{i+1}^2 - \sigma_i^2 = 1.69$. A particle at $x = 5$ has score $-5/101 = -0.0495$, so the drift moves it $1.69 \times 0.0495 = 0.084$ toward zero, and the noise adds a kick with standard deviation $\sqrt{1.69} = 1.30$. Each single step is mostly noise; the inward drift adds up over hundreds of steps and shrinks the spread from 50 at the prior to 1 at the data.

Figure 2 draws the score of a noised mixture of six clusters. Move the noise level and watch where the arrows point.

Generation therefore needs $\nabla_{\mathbf{x}}\log p_t(\mathbf{x})$ at every $t$, sharp near the end of generation and smooth at the start, estimated from samples of the data alone.

Learning the score by denoising

Nobody has ground-truth score vectors to regress on, since $p_t$ is unknown. Score matching (Hyvärinen, 2005) gives an objective whose minimizer is the score anyway, and its denoising form (Vincent, 2011) needs nothing but noised training pairs. The paper trains a time-conditioned network $\mathbf{s}_{\boldsymbol\theta}(\mathbf{x}, t)$ with

Read from the inside out: draw a clean sample $\mathbf{x}(0)$ from the data, draw $t$ uniformly from $[0, T]$, noise the sample with the forward process to get $\mathbf{x}(t)$, and regress the network's output onto the score of the transition kernel $p_{0t}(\mathbf{x}(t) \mid \mathbf{x}(0))$, the distribution of $\mathbf{x}(t)$ given that particular starting point. $\lambda(t) > 0$ weights the noise levels.

The target is not the score of $p_t$, so why does the minimizer equal it? The noised distribution is a mixture, one kernel per data point, averaged:

The gradient of the log of a mixture is the average of the components' gradients, each weighted by the posterior probability that this component produced $\mathbf{x}$:

So each training target is a noisy sample whose conditional mean, at that $\mathbf{x}(t)$, is the true score, and the squared-error regression converges to that conditional mean. Vincent showed the per-pair objective and the objective with the true score differ by a constant that does not depend on $\boldsymbol\theta$, so they have the same gradients. At the optimum the network equals $\nabla_{\mathbf{x}}\log p_t(\mathbf{x})$ for almost all $\mathbf{x}$ and $t$. For Gaussian kernels the identity has a geometric reading (Tweedie's formula): $\sigma^2$ times the score of the noised data points from $\mathbf{x}$ to $\mathbb{E}[\mathbf{x}_0 \mid \mathbf{x}]$, the average clean point that could have produced it.

Figure 3 shows the averaging with seven clean points. Drag the white probe to a noisy location, then press draw a training pair a few times to see single targets.

The target is cheap whenever the drift $\mathbf{f}$ is affine in $\mathbf{x}$, because the kernel is then Gaussian with a closed-form mean $\boldsymbol\mu_t(\mathbf{x}_0)$ and standard deviation $\sigma_t$. The score of $\mathcal{N}(\boldsymbol\mu, \sigma^2 I)$ at $\mathbf{x}$ is $(\boldsymbol\mu - \mathbf{x})/\sigma^2$, with the variance $\sigma^2$ in the denominator. Writing $\mathbf{x}(t) = \boldsymbol\mu + \sigma\mathbf{z}$, the target is $-\mathbf{z}/\sigma$, the drawn noise divided by the noise level.

The weighting fixes a scale problem. The expected squared norm of that target is $d/\sigma^2$ for $d$ dimensions. On CIFAR-10 ($d = 3072$) with noise levels from 0.01 to 50, that runs from $3.07 \times 10^7$ down to 1.23, a spread of seven orders of magnitude, and an unweighted loss would spend almost all its gradient on the smallest noise levels. The paper sets $\lambda(t) \propto 1/\mathbb{E}\lVert\nabla\log p_{0t}\rVert^2$, which here is $\lambda = \sigma^2$, and the loss becomes $\lVert \sigma\,\mathbf{s}_{\boldsymbol\theta} + \mathbf{z}\rVert^2$, a quantity of order $d$ at every noise level. The official code computes exactly that:

# one training step of the score network (VE SDE, CIFAR-10 settings)
x0 = next(loader)                     # clean images, shape (128, 3, 32, 32)
t = rand(128) * (1 - 1e-5) + 1e-5     # one time per image, t ~ U(1e-5, 1)
mean, std = marginal_prob(x0, t)      # VE: mean = x0, std ~ 0.01 * 5000**t
z = randn_like(x0)
x_t = mean + std[:, None, None, None] * z        # noised images, same shape
s = score_net(x_t, t)                 # predicted score, shape (128, 3, 32, 32)
# target score of the kernel is -z / std; lambda(t) = std**2 turns
# lambda * |s - (-z/std)|^2 into |s * std + z|^2
loss = ((s * std[:, None, None, None] + z) ** 2).mean()
loss.backward()                       # gradient reaches score_net's weights only

Training has no sampling loop, no adversary and no chain to unroll: one noise draw, one forward pass and one squared error per image. The objective works for any SDE with an affine drift, and the paper notes that sliced score matching (Song et al., 2019) can replace the denoising target when the kernel has no closed form.

VE, VP and sub-VP: SMLD and DDPM as two SDEs

Choosing $\mathbf{f}$ and $g$ chooses a model family. The paper takes the noising chains of SMLD and DDPM, lets their number of steps $N$ go to infinity, and reads off the SDE each one becomes.

SMLD noises a clean sample by adding independent Gaussian noise of growing size and never rescales the signal:

with $\sigma_0 = 0$, so after $i$ steps the total added variance is $\sigma_i^2$. As $N \to \infty$ the chain becomes an SDE with no drift:

Nothing pulls the signal back, so the variance grows with $\sigma(t)$ without bound, and the paper names (9) the Variance Exploding (VE) SDE. In the experiments $\sigma(t) = \sigma_{\min}(\sigma_{\max}/\sigma_{\min})^t$ with $\sigma_{\min} = 0.01$ and, for CIFAR-10, $\sigma_{\max} = 50$. The kernel has mean $\mathbf{x}_0$ and variance $\sigma(t)^2$ per coordinate (Eq. 31), and the prior is $\mathcal{N}(0, \sigma_{\max}^2 I)$, wide enough that pixel values in $[0, 1]$ are lost in it. Because $\sigma(t)$ tends to $\sigma_{\min} = 0.01$ as $t \to 0$ while the chain starts from $\sigma_0 = 0$, the VE SDE is undefined at $t = 0$, so the paper solves it on $[10^{-5}, 1]$; that is the 1e-5 in the training code above.

DDPM shrinks the signal at every step as it adds noise:

Scaling by $\sqrt{1 - \beta_i}$ multiplies the variance by $1 - \beta_i$, the fresh noise adds $\beta_i$, and the two sum to 1, so unit-variance data keeps unit variance at every step. Write $\beta_i = \bar\beta_i / N$, where $\bar\beta_i = \beta(i/N)$ samples a continuous schedule $\beta(t)$ on $[0, 1]$; as $N \to \infty$ the limit is

the Variance Preserving (VP) SDE. The drift $-\tfrac{1}{2}\beta(t)\mathbf{x}$ pulls toward the origin at the rate that offsets the injected noise. The paper uses $\bar\beta_{\min} = 0.1$ and $\bar\beta_{\max} = 20$ as the endpoints of a linear $\beta(t)$, which is the $\beta(t) = 0.1 + 19.9\,t$ used above. With $N = 1000$ these give DDPM's own schedule of $\beta_1 = 10^{-4}$ rising linearly to $\beta_N = 0.02$. Its kernel has mean $\mathbf{x}_0 e^{-\frac{1}{2}\int_0^t \beta}$ and variance $1 - e^{-\int_0^t \beta}$, and its prior is $\mathcal{N}(0, I)$. The names VE and VP are this paper's; neither SMLD nor DDPM used them.

The paper also proposes a new SDE, the sub-VP SDE. It keeps the VP drift and scales down the noise term:

The extra factor is 0 at $t = 0$ and approaches 1 as $\int\beta$ grows, so sub-VP has the same mean as VP, injects less noise early, and reaches the same $\mathcal{N}(0, I)$ prior. The paper proves its variance is at most the VP variance at every time. For unit-variance data, writing $u = e^{-\int_0^t\beta}$, Appendix B's formula gives

which is 1 at both ends ($u = 1$ and $u \to 0$) and has its minimum, 0.75, at $u = 1/2$. That dip is our derivation from the paper's formula, not a number the paper prints. In experiments sub-VP always gave higher likelihoods than VP; the paper reports this result without proposing a mechanism for it. Scrub Figure 4 to compare the three variances.

With one codebase, swapping the forward SDE changes only the kernel's mean and standard deviation in training and the drift and diffusion in sampling. The paper uses that to compare SMLD-style and DDPM-style models with the same samplers and to find better architectures for each (NCSN++ for VE, DDPM++ for VP and sub-VP), which is how SMLD, previously well behind DDPM on FID, catches up with it.

The probability-flow ODE

The reverse SDE adds fresh noise at every step, so two runs from the same prior sample end at different images. The paper shows that every diffusion of the form (5) has a deterministic counterpart, an ordinary differential equation, whose solutions have the same marginal densities $p_t$ at every time:

The paper calls it the probability flow ODE. Compared with the reverse SDE (6), the noise term is gone and the score's coefficient is exactly half, $\tfrac{1}{2}g(t)^2$ instead of $g(t)^2$. Individual trajectories differ from the SDE's; the distribution of particles at each $t$ is the same.

Both coefficients come out of the Fokker-Planck equation, which describes how the density $p_t$ of an SDE evolves. For (5) with scalar $g$, it reads

The first term moves probability along the drift; the second spreads it. The $\tfrac{1}{2}$ comes from the second-order Taylor term in Itô's formula, applied to a Wiener increment whose variance is $dt$. Since $\nabla^2 p = \nabla\cdot(p\,\nabla\log p)$, the spreading term can be rewritten as a flow:

This is the continuity equation for particles moving deterministically with velocity $\mathbf{f} - \tfrac{1}{2}g^2\nabla\log p_t$, which is the ODE (13); Appendix D.1 of the paper does this calculation in general form, an idea it credits to Maoutsa et al. (2020). The reverse SDE keeps a noise term, and in reversed time that noise spreads the density by $+\tfrac{1}{2}g^2\nabla^2 p_t$ while the reversed process has to contract it by the same amount. The drift has to cancel the noise's spreading and also supply the contraction, two halves, so its score coefficient is the full $g^2$ of Anderson's result. The argument checks that (6) and (13) agree with each other; Anderson (1982) has the rigorous proof of (6).

Figure 5 runs both on a one-dimensional mixture, where $p_t$ is known exactly. The upper panel integrates the reverse SDE, the lower one the ODE, from the same 3000 prior draws with 400 steps. Play it, then switch to the swapped coefficients.

With a learned score in place of the true one, (13) is a neural ODE (Chen et al., 2018): a deterministic, invertible map from data to prior whose velocity is a network, and four properties follow from that.

First, the ODE gives exact likelihoods. As the flow carries a small volume of probability, the density inside changes by the divergence of the velocity $\tilde{\mathbf{f}} = \mathbf{f} - \tfrac{1}{2}g^2\mathbf{s}_{\boldsymbol\theta}$ (the instantaneous change-of-variables formula):

Take 1D data $\mathcal{N}(0, 1)$ under the VE SDE. With $\mathbf{f} = 0$ and $g^2\,dt = d[\sigma^2]$, the ODE (13) becomes, after the $\tfrac{1}{2}$ cancels against the 2 in $d[\sigma^2] = 2\sigma\,d\sigma$,

a form the paper does not print but which follows from (13) directly. DDIM's deterministic sampler is an Euler step of this equation in rescaled coordinates, and Karras et al. (2022, EDM) write their ODE in this form. Here the score is $-x/(1 + \sigma^2)$, so $dx/d\sigma = \sigma x/(1 + \sigma^2)$ and $x$ is proportional to $\sqrt{1 + \sigma^2}$. The data point $x(0) = 1$ flows to $x(50) = \sqrt{2501} = 50.01$, one standard deviation of the data to one standard deviation of the prior. The divergence $\sigma/(1 + \sigma^2)$ integrates to $\tfrac{1}{2}\ln 2501 = 3.912$ nats. The prior density at 50.01 is $\log \mathcal{N}(50.01; 0, 2501) = -5.331$, and adding 3.912 gives $-1.419$, which is $\log \mathcal{N}(1; 0, 1)$ exactly. A VAE's ELBO only bounds this number from below. The divergence of a real network over 3072 dimensions is too expensive to compute exactly, so the paper estimates it with the unbiased Skilling-Hutchinson trace estimator and averages runs to reach the needed accuracy: exact in the sense of no bound, but computed with random probes.

Second, the encoding is unique. The forward SDE has no trainable parameters, so for a given data distribution, integrating (13) forward maps each data point to one latent in the prior, the same for any model that learns the score accurately. Third, mixing two latents and decoding gives images in between, the interpolation familiar from flows and VAEs. Fourth, a black-box ODE solver (the paper uses RK45) picks its own step sizes and cuts score-network evaluations by more than 90% without visible loss in sample quality.

Predictor-corrector sampling

A discretized reverse SDE carries error: after each finite step the samples are distributed slightly differently from $p_t$, and the errors accumulate. Smaller steps reduce the error at a proportional cost. The paper adds a different kind of step that uses the score to pull the samples back toward $p_t$ at a fixed noise level.

That step is Langevin dynamics, a Markov chain Monte Carlo method that needs only the score of its target. In this paper's convention, one Langevin step at level $t$ is

The score term alone is gradient ascent on $\log p_t$ and would pull every sample onto a mode. The noise term spreads them out. With step $\epsilon$ on the drift and $\sqrt{2\epsilon}$ on the noise, the two balance so that, as $\epsilon \to 0$ and the number of steps grows, the samples are distributed as $p_t$ (under regularity conditions). The $\sqrt{2\epsilon}$ pairing is specific to this convention; the NCSN paper writes the same dynamics as $\tfrac{\alpha}{2}\nabla\log p + \sqrt{\alpha}\,\mathbf{z}$, so its $\alpha$ equals $2\epsilon$ here.

A finite step does not land exactly on $p_t$. For a Gaussian mode of variance $s^2$, the update is

and the variance $v$ it settles at solves

larger than $s^2$ for any $\epsilon > 0$, and unbounded once $\epsilon \ge 2s^2$. Small steps have small bias but need many iterations to move the samples; large steps move fast and settle on the wrong spread. Figure 6 starts from a predictor output packed too tightly around each mode. Raise $M$, the number of Langevin steps, then try the smallest and largest $\epsilon$.

A predictor-corrector (PC) sampler alternates the two. The predictor is any numerical solver of the reverse SDE and takes one step down in time; the corrector runs $M$ Langevin steps at the new time with the same score network. The earlier methods are special cases: SMLD is an identity predictor (no time step from the SDE) with annealed Langevin as the corrector, and DDPM is its ancestral sampler as the predictor with an identity corrector. DDPM's ancestral sampling is one particular discretization of the reverse VP SDE; the paper's reverse-diffusion predictor discretizes it the way the forward SDE is discretized and does slightly better.

The corrector's step size comes from a signal-to-noise ratio $r$ (0.16 for VE on CIFAR-10), with norms averaged over the batch:

If the network's score has the size of the kernel score, $\lVert\mathbf{s}\rVert \approx \sqrt{d}/\sigma$, while $\lVert\mathbf{z}\rVert \approx \sqrt{d}$, so $\epsilon \approx 2r^2\sigma^2$, which is $0.0512\,\sigma^2$: the step scales with the noise variance, which is how NCSN set its Langevin steps. The full loop for VE, as in the paper's Algorithm 2:

# PC sampling for the VE SDE (Algorithm 2), N = 1000, M = 1
x = randn(shape) * sigma[N]                    # prior N(0, 50^2 I)
for i in range(N - 1, -1, -1):                 # noise level runs down to sigma[0]
    # predictor: one reverse-diffusion step of Eq. (6)
    d = sigma[i + 1] ** 2 - sigma[i] ** 2
    x = x + d * score_net(x, sigma[i + 1]) + sqrt(d) * randn_like(x)
    # corrector: M Langevin steps at the new level, step size from r = 0.16
    for _ in range(M):
        g = score_net(x, sigma[i])
        z = randn_like(x)
        eps = 2 * (r * norm(z) / norm(g)) ** 2
        x = x + eps * g + sqrt(2 * eps) * z
return x

Table 1 of the paper compares samplers on CIFAR-10 at 1000 and 2000 score evaluations. For the VE model with the reverse-diffusion predictor, 1000 predictor steps give FID 4.79, 2000 give 4.74, 2000 corrector-only steps give 20.43, and PC1000 (1000 of each, 2000 evaluations) gives 3.60. For the VP model the same four are 3.21, 3.19, 19.06 and 3.18. Adding one corrector step per predictor step always improved on P1000 and was typically better than doubling the predictor steps; corrector-only was worst in both columns. With the probability-flow ODE as the predictor, the VE FID drops from 15.41 to 3.51 when correctors are added.

Conditional generation: class labels, inpainting, colorization

To sample from $p(\mathbf{x} \mid \mathbf{y})$ for some condition $\mathbf{y}$, the reverse SDE needs the conditional score. Bayes' rule gives

and the last term has no $\mathbf{x}$ in it, so its gradient vanishes. The conditional reverse SDE is

The first gradient is the unconditional score already trained. The second is the gradient of how likely the condition is given the noisy sample, and both are multiplied by the same $g(t)^2$. Equation (14) has no extra scale factor on the second term; the guidance scale came later, with Dhariwal and Nichol (2021).

Where the second gradient comes from depends on the task. For class labels, $p_t(\mathbf{y} \mid \mathbf{x})$ is a classifier that has to work on noised inputs, so the paper trains a separate time-dependent classifier on samples from the forward SDE, with a cross-entropy loss mixed over time steps. The score model is not retrained. For inpainting (imputation) and colorization the paper needs no auxiliary model: the known pixels are replaced at each step by a copy of their true values noised with the forward kernel to the current time, and the unconditional score of the combined image moves the unknown pixels along the reverse SDE. Appendix I.2 derives this as an approximation to the conditional score. Colorization is the same imputation in an orthogonally transformed color space. The demonstrations are class-conditional samples on 32×32 CIFAR-10 and inpainting and colorization on 256×256 LSUN.

In Figure 7, pick a class. The unconditional score sends particles to all three clusters; selecting a class adds the gradient of a classifier's log-probability to the same score.

Every class-conditional sampling step costs a backward pass through the classifier, to get $\nabla_{\mathbf{x}}\log p_t(\mathbf{y} \mid \mathbf{x})$, on top of the score evaluation, and the classifier has to be accurate on images at every noise level. Classifier-free guidance later removed the separate classifier by training one network with and without the condition.

Results and limits

The best sample quality came from NCSN++ cont. (deep, VE): on unconditional CIFAR-10, an Inception score of 9.89 and an FID of 2.20, with PC samplers. The Inception score rewards samples that a classifier labels confidently and that cover many classes (higher is better). The path there was incremental: NCSN++ with PC sampling reached 2.45, training it with the continuous objective (7) brought 2.38, and doubling the residual blocks per resolution ("deep") brought 2.20.

An FID of 2.20 without labels beat the best class-conditional model then reported, StyleGAN2-ADA at 2.42, on FID. The conditional GAN's Inception score of 10.14 is still higher than 9.89; unconditional StyleGAN2-ADA scores 2.92 and 9.83.

The best likelihood came from a different model, DDPM++ cont. (deep, sub-VP): 2.99 bits per dimension, computed with the probability-flow ODE, with an ODE-sampled FID of 2.92. Bits per dimension is the average number of bits the model needs to encode one pixel channel of a test image, so lower means test images are more probable under the model. The paper calls 2.99 a record for uniformly dequantized CIFAR-10, a comparison that leaves out models evaluated with variational dequantization or on discrete data, and it was reached without maximum-likelihood training. The two sets of numbers also use different checkpoints: Table 3 (2.20 and 9.89) reports the checkpoint with the lowest FID during training, sampled with PC; Table 2 (2.99 and 2.92) reports the last checkpoint, sampled with a black-box ODE solver. VE SDEs gave better samples and worse likelihoods than VP and sub-VP, so the choice of SDE depends on which number matters.

At higher resolution, an NCSN++-style model trained with (7) produced 1024×1024 CelebA-HQ faces, sampled with 2000 PC steps, the first high-fidelity images at that size from a score-based model. The authors note the samples are not perfect, with visible flaws in facial symmetry.

The paper names sampling speed as its main limitation: PC1000 calls the score network 2000 times per sample, and the 1024×1024 sampler runs 2000 predictor steps plus a corrector step after each, where a GAN generates with one forward pass. The paper leaves combining stable score-model training with fast sampling to future work, and lists the samplers' many hyperparameters (step counts, signal-to-noise ratios) as a second limitation. The follow-ups split along those lines: deterministic samplers built on the ODE, such as DDIM, cut the step count; Flow Matching trains an ODE's velocity field directly; and Latent Diffusion runs the same machinery in a compressed latent space. DiffusionBlocks reads the blocks of an ordinary residual network as steps of a reverse diffusion.