Consistency Models

Map any noisy image straight to the clean image its sampling path would reach, and a diffusion model samples in one network call.

A consistency model learns the endpoint of a diffusion model's sampling trajectory as a function of any point on it. It can be distilled from a trained diffusion model or trained from scratch, and it samples in one to four network evaluations.

The same sampler that generates an image also inpaints, colorizes and upsamples one, with no task-specific training.

A diffusion model generates by iterating. A network estimates the clean image hiding under a noisy one, the sampler removes a little of the noise, and the loop repeats. The best sampler for the diffusion models this paper starts from (EDM, Karras et al., 2022) needs 35 network evaluations per image on CIFAR-10 and 79 on ImageNet 64×64, each a full pass through a U-Net with tens of millions of parameters. A GAN produces an image in one pass, but its training is adversarial, and it has no way to spend more compute for a better sample and no built-in route to inpainting, colorization or super-resolution.

Consistency models (Song, Dhariwal, Chen and Sutskever, OpenAI, 2023) keep the diffusion framework and get one-step generation out of it. Run a diffusion model's sampler deterministically and it traces a smooth curve from a noise vector to an image. A consistency model is a network trained to return that curve's endpoint from any point along it. Given pure noise it returns the finished image in one call; given a lightly noised image it returns that image cleaned. Every point on one curve has to map to the same endpoint, and the paper names that property self-consistency.

The page follows the paper's order: the deterministic sampling curve and why it exists; the consistency function and the one hard constraint a network must satisfy to represent it; sampling in one step and in a few; the two training methods, distilling a trained diffusion model and training from scratch with no teacher; and the numbers, which include a one-step FID of 3.55 on CIFAR-10 against 8.34 for the previous best distillation method.

Sampling a diffusion model is solving an ODE

The paper uses the conventions of EDM (Karras et al., 2022). An image $\mathbf{x}$ has pixels in $[-1, 1]$. Noising adds Gaussian noise of standard deviation $t$, with nothing subtracted from the signal:

So $t$ is both the time variable and the noise level. At $t = 80$ the pixels, whose magnitude is at most 1, sit under noise with standard deviation 80, and the distribution of $\mathbf{x}_T$ is indistinguishable from $\mathcal{N}(\mathbf{0}, 80^2\mathbf{I})$. At $t = \epsilon = 0.002$ the image is clean to well below one grey level out of 256. Write $p_t$ for the distribution of $\mathbf{x}_t$: the data distribution blurred by a Gaussian of width $t$.

Noising is a random process. The result the paper builds on, from Song et al. (2021), is that there is also a deterministic process with exactly the same distribution $p_t$ at every $t$: the probability-flow ODE. For the noising in (1) it reads

The middle form is the paper's Equation 3 with the exact score in place of the learned one. The gradient $\nabla\log p_t$ is the score, the direction in which noisy images become more probable, and Tweedie's formula rewrites it through the denoiser $D(\mathbf{x}_t, t) = \mathbb{E}[\mathbf{x}\mid\mathbf{x}_t]$, the average clean image that could have produced $\mathbf{x}_t$: the score equals $(D - \mathbf{x}_t)/t^2$. A diffusion model is a network trained to estimate $D$ (or, equivalently, the score), and the DiffusionBlocks explainer walks that identity with a probe you can drag. The right-hand form of (2) is what the code computes: the velocity points from the current noisy image toward its denoised estimate, scaled by $1/t$.

An ODE has one solution through each starting point, so the noise vectors at $t = 80$ are matched one to one with images at $t = \epsilon$ and the solution curves never cross. To sample, draw $\hat{\mathbf{x}}_T\sim\mathcal{N}(\mathbf{0}, T^2\mathbf{I})$ and integrate (2) downward in $t$ with a numerical solver. EDM uses Heun's second-order method (two denoiser calls per step) on a grid of noise levels that is dense near the data and sparse near pure noise:

With $N = 18$ the grid runs 0.002, 0.008, 0.023, 0.060, 0.140, 0.296, 0.585, 1.088, 1.923, 3.257, 5.315, 8.401, 12.9, 19.4, 28.4, 40.8, 57.6, 80. EDM's sampler takes 18 Heun steps from 80 down to 0 (the last step, onto zero, is a plain Euler step), which is 35 denoiser calls, the CIFAR-10 teacher's 35 NFE (network function evaluations).

The figures on this page use a one-dimensional stand-in for images so the ODE can be integrated exactly: the data is a mixture of three narrow Gaussians at $-1.2$, $0.2$ and $1.3$, with the paper's $\epsilon$, $T$, $\sigma_{\text{data}} = 0.5$ and grid. Each plot runs noise at the top to data at the bottom, with the vertical axis warped so the grid (3) is evenly spaced, and the horizontal axis showing $\mathbf{x}_t$ divided by its typical size $\sqrt{t^2 + 0.25}$, so the band of plausible points has the same width at every height. That division is also what the network's input scaling does, in the next section.

The consistency function

Take one solution curve $\{\mathbf{x}_t\}_{t\in[\epsilon,T]}$ of (2). The consistency function sends every point on it to the curve's origin:

The first equality is self-consistency: two points on the same curve give the same answer. The second is the boundary condition: at the smallest noise level the function is the identity, because a point at $t = \epsilon$ is its own origin. Both are consequences of the definition, and the training methods below enforce the first while the architecture enforces the second.

Drag the probe in Figure 1 anywhere in the plane. The trajectory through it lights up, and the dashed chord drops from the probe to the trajectory's origin. Now slide the probe up or down along that same trajectory and watch the readout $f(x_t, t)$: it does not move. Then step sideways onto a neighbouring trajectory and it jumps to a different origin.

The function takes both the point and its noise level, and the figure shows why it needs both: at a fixed $t$, nearby points on different trajectories land at different origins, while the same point at a different $t$ would belong to a different trajectory altogether.

Compare $\mathbf{f}$ with the denoiser $D$ from (2). Both take a noisy image and a noise level and return a clean image, and near $t = \epsilon$ they agree, since both return roughly the input. At high noise they answer different questions. $D(\mathbf{x}_T, T)$ is the average of every image that could sit under that noise, which for the toy is the mixture mean $0.15$ and for CIFAR-10 is a grey blur. $\mathbf{f}(\mathbf{x}_T, T)$ is the origin of the one trajectory through $\mathbf{x}_T$: for the probe starting at $\mathbf{x}_T = 48$ in Figure 1 it is $1.23$, a sample from the right-hand mode. A network that fits $\mathbf{f}$ is the whole ODE solver in one call, and a network that fits $D$ is one solver step.

The boundary condition also rules out a cheat. A constant function $\mathbf{f}\equiv\mathbf{0}$ is perfectly self-consistent, since every point on every trajectory maps to the same output. It violates $\mathbf{f}(\mathbf{x}_\epsilon,\epsilon) = \mathbf{x}_\epsilon$.

Wiring the boundary condition into the network

A free-form network $F_\theta(\mathbf{x}, t)$ will not return $\mathbf{x}$ exactly at $t = \epsilon$ on its own. The paper gives two fixes. The first is a hard switch: output $\mathbf{x}$ when $t = \epsilon$ and $F_\theta$ otherwise. The second, which every experiment uses, blends the input and the network output with two scalar functions of $t$:

Any differentiable pair with those two values works, and the paper takes EDM's pair with $t$ replaced by $t - \epsilon$:

EDM's originals, without the shift, give $c_{\text{skip}}(\epsilon) = 0.999984$ and $c_{\text{out}}(\epsilon) = 0.0020$: the network's output leaks into the answer at the two-tenths-of-a-percent level right where the identity has to be exact. The shift costs nothing and closes the gap. Slide $t$ in Figure 2 to the left end and toggle the comparison to read both pairs; at any other $t$ the two are visually the same curve.

The code keeps two more EDM conventions. The image is multiplied by $c_{\text{in}}(t) = 1/\sqrt{t^2 + \sigma_{\text{data}}^2}$ before the first layer, so at $t = 80$ an input of size 80 arrives as a unit-variance tensor ($c_{\text{in}} = 0.0125$) and at $t = 1.088$ it is scaled by 0.835. The noise level itself enters the network's timestep embedding as $250\ln t$, which is EDM's $\ln(t)/4$ multiplied by 1000 to suit the embedding layer inherited from ADM (Dhariwal and Nichol, 2021); at $t = 1$ that number is 0, at $t = 80$ it is 1096.

Because (5) has the same shape as EDM's own output parameterization, a consistency model can reuse a diffusion model's architecture unchanged: NCSN++ on CIFAR-10 and the ADM U-Nets on ImageNet 64×64 and LSUN 256×256. Distillation starts from the teacher's weights, so at initialization $\mathbf{f}_\theta$ already computes the teacher's denoiser and only has to learn to jump further.

One step, or a few: jump and renoise

One-step sampling is the definition: draw $\hat{\mathbf{x}}_T\sim\mathcal{N}(\mathbf{0}, T^2\mathbf{I})$ and return $\mathbf{f}_\theta(\hat{\mathbf{x}}_T, T)$. For a better sample, Algorithm 1 alternates two moves. Jump: apply $\mathbf{f}_\theta$ to get a clean estimate. Renoise: add fresh Gaussian noise to bring the estimate to a chosen level $\tau_n$, lower than the previous one, then jump again.

# Algorithm 1: multistep consistency sampling (N calls)
x = f(T * randn(3, 32, 32), T)          # one-step sample, T = 80
for tau in taus:                          # e.g. taus = [1.38] for N = 2
    z  = randn(3, 32, 32)
    xt = x + sqrt(tau**2 - eps**2) * z    # renoise to level tau
    x  = f(xt, tau)                       # jump again
return x

The noise added is $\sqrt{\tau_n^2 - \epsilon^2}\,\mathbf{z}$, because the jump's output lives at level $\epsilon$, not zero; adding variance $\tau_n^2 - \epsilon^2$ on top of $\epsilon^2$ gives a point at exactly $\tau_n$, where the model was trained. Numerically the correction is invisible ($\sqrt{1.382^2 - 0.002^2} = 1.381999$), but the bookkeeping is the same one the code does. The paper picks the levels $\tau_1 > \tau_2 > \cdots$ by a greedy ternary search on FID, one level at a time, on the assumption that FID is unimodal in the next level given the previous ones. The released two-step ImageNet 64×64 distillation checkpoint renoises to $\tau_1 = 1.38$, which is index 22 of a 40-point grid (3); the LSUN 256 one uses $\tau_1 = 4.37$, and the from-scratch ImageNet model $1.91$.

Figure 3 runs Algorithm 1 on the toy with an imperfect student: its output is the exact $\mathbf{f}$ pulled toward the blurry denoiser $D$ by a fraction $0.55\sqrt{t/T}$, so it is nearly exact at low noise and half-blurred at $t = 80$, which is the shape of error a real one-step model shows. The strip at the bottom compares where 400 particles land with the data density, and the readout is the distance between the two (the 1-D Wasserstein distance, the average shift needed to move one histogram onto the other). Start at $N = 1$, then go to 2 and watch the landings pull onto the modes; then, at $N = 2$, sweep $\tau_1$ from the bottom of its range to the top.

The second step helps because it is taken from a level where the student is accurate. Renoise too little ($\tau_1$ near 0.1) and the point stays on top of the first jump's error, so the second jump repeats it. Renoise too much ($\tau_1$ near 80) and the second jump is as blurry as the first. The minimum sits in between, near where the paper's search put it. The paper reports the same trend on real data: CIFAR-10 FID drops from 3.55 to 2.93 going from one step to two, ImageNet from 6.20 to 4.70.

The same loop does zero-shot editing (Algorithm 4 in the appendix). For inpainting, start from the reference image with the hole zeroed, noised to the first level; after every jump, overwrite the known pixels with the reference and keep the model's output only inside the mask, then renoise the whole image to the next level and jump again. Colorization and super-resolution use the same replacement through an invertible linear transform (a grey-scale projection, a downsampling), stroke-guided generation noises a hand-drawn sketch to some $t$ and jumps, as in SDEdit, and interpolation between two images is interpolation between their noise vectors followed by one jump, since $\mathbf{f}_\theta(\cdot, T)$ is a deterministic map. None of these tasks appears in training.

Consistency distillation: pairs from a teacher's step

Training needs a signal for $\mathbf{f}$. The direct route, solving the ODE all the way to $\epsilon$ for every training noise vector and regressing onto the result, costs a full 35-call sampler per example; earlier distillation methods (knowledge distillation, DFNO) pay it by generating a synthetic dataset first. Consistency distillation instead enforces self-consistency between adjacent points on a trajectory, which needs only one solver step.

Discretize $[\epsilon, T]$ with the grid (3), $t_1 = \epsilon < \cdots < t_N = T$. Draw an image $\mathbf{x}$ from the dataset, an index $n$ uniformly from $\{1,\dots,N-1\}$, and a noisy point $\mathbf{x}_{t_{n+1}} = \mathbf{x} + t_{n+1}\mathbf{z}$. The trained diffusion model $\mathbf{s}_\phi \approx \nabla\log p_t$ (equivalently its denoiser $D_\phi$) supplies the velocity, and one solver step lands an estimate of the same trajectory at $t_n$:

$t_n - t_{n+1}$ is negative, so the step moves down in noise. The experiments use Heun instead of Euler: evaluate the velocity at the Euler guess too and average the two, at the price of a second teacher call. The pair $(\mathbf{x}_{t_{n+1}}, \hat{\mathbf{x}}^{\phi}_{t_n})$ should map to the same origin, and the loss penalizes the difference:

$d$ is a distance, $\lambda \equiv 1$ in every experiment, and the two ends use different weights. The online network $\theta$ sees the noisier point and receives the gradient. The target network $\theta^-$ sees the less noisy point, with the gradient blocked, and its weights start as a copy of the online ones and then follow them as an exponential moving average:

The names come from deep Q-learning and from momentum-based self-supervised learning such as BYOL and MoCo (the I-JEPA explainer has a figure on what $\mu$ does). Why not let both ends move? With $\theta^- = \theta$ and gradient through both, gradient descent shrinks the loss fastest by moving both outputs toward each other, or toward a constant, with nothing tying them to the trajectory; the boundary condition prevents the collapse but not the drift. The paper reports that the EMA plus stopgrad target "greatly" stabilizes training and improves the final model. On CIFAR-10 distillation uses $\mu = 0$, so the target is the current online weights with the gradient cut; on ImageNet and LSUN $\mu = 0.95$.

One CIFAR-10 step in full, with $N = 18$, Heun, LPIPS and a batch of 512:

# one consistency-distillation step, CIFAR-10 settings (N = 18, Heun)
x  = batch()                          # [512, 3, 32, 32], pixels in [-1, 1]
n  = randint(1, N - 1)                # say n = 7: t7 = 0.585, t8 = 1.088
z  = randn_like(x)
x_hi = x + t[n+1] * z                 # the online network's input
d1 = (x_hi - D_teacher(x_hi, t[n+1])) / t[n+1]        # ODE velocity
x_eu = x_hi + (t[n] - t[n+1]) * d1                     # Euler guess
d2 = (x_eu - D_teacher(x_eu, t[n])) / t[n]
x_lo = x_hi + (t[n] - t[n+1]) * 0.5 * (d1 + d2)        # Heun step (7)
loss = lpips(f_online(x_hi, t[n+1]), f_target(x_lo, t[n]).detach())
loss.backward()                       # gradient reaches theta only
opt.step()
theta_target = mu * theta_target + (1 - mu) * theta    # (9); mu = 0 here

Per iteration that is one forward and backward pass of the online network, one forward of the target, and two forwards of the teacher. The Learned Perceptual Image Patch Similarity (LPIPS) distance compares the two outputs through a pretrained feature network, after upsampling the 32×32 images to 224×224; the paper also tried squared $\ell_2$ and $\ell_1$, and LPIPS beat both by a wide margin at every point in training (its Figure 3a).

Figure 4 shows what the teacher's step contributes. The filled dot is $\mathbf{x}_{t_{n+1}}$, the hollow ring is where the exact ODE would put its partner at $t_n$, the teal dot is the solver step (7), and the amber dot is a third candidate the next section uses. Read the header: the teal point's distance from the exact one is the solver's local error, and the "endpoint gap" is how far apart the two points' true origins are. Start at $N = 9$ with Euler, switch to Heun, then drag $N$ to 80.

The teacher's information enters the student only through that teal point, and Theorem 1 quantifies the consequence. If the distillation loss reaches zero with $\theta^- = \theta$, the model is Lipschitz, and the solver's local error is $O(\Delta t^{p+1})$ with $\Delta t$ the largest grid gap, then

Euler has $p = 1$ and Heun $p = 2$, so at equal $N$ Heun's bound is one order tighter, and the proof is the standard global-error argument for ODE solvers: the per-pair errors accumulate across the $N - 1$ intervals, times the Lipschitz constant. The experiments agree: at every $N$ Heun beats Euler, $N = 18$ is best on CIFAR-10, and above that the FID stops moving with $N$. The $\mathbf{f}(\cdot;\phi)$ on the right of (10) is the consistency function of the teacher's ODE, so a distilled model inherits the teacher's errors along with its trajectories.

Consistency training: the same pairs with no teacher

Distillation needs $\mathbf{s}_\phi$ to build the partner point. The paper's Lemma 1 gives an estimate of the score that needs no network. Since $\mathbf{x}_t = \mathbf{x} + t\mathbf{z}$,

is an unbiased but noisy estimate of the score at $\mathbf{x}_t$, using only the clean image the point was made from. Put it into the Euler step of (7) in place of $\mathbf{s}_\phi$ and every term is known:

The estimated step lands on the same noise vector at the lower level. Nothing has to be solved: the partner point is the amber dot in Figure 4, on the straight ray $\mathbf{x} + t\mathbf{z}$. Substituting it into (8) gives the consistency training loss:

Theorem 2 relates the two losses: with the Euler solver and a teacher whose score is exact, $\mathcal{L}_{\text{CD}}^N = \mathcal{L}_{\text{CT}}^N + o(\Delta t)$, and $\mathcal{L}_{\text{CT}}^N$ itself is at least $O(\Delta t)$ whenever the distillation loss does not vanish. Consistency training is therefore distillation from a perfect Euler teacher, up to terms that shrink faster than the loss does as the grid gets finer. Figure 4 shows the geometry: the amber point is off the trajectory by more than the teal one at small $N$, and both gaps close as $N$ grows. Because no solver is involved, there is no Heun to switch on; the grid size $N$ is the only accuracy setting.

Choosing $N$ trades off two things, and the paper handles it with a schedule rather than one setting. At $N = 2$ the grid is $\{0.002, 80\}$, the target network receives $\mathbf{x} + 0.002\mathbf{z}$ at $t = 0.002$ and returns it unchanged by the boundary condition, and (13) is a plain regression from $\mathbf{x} + 80\mathbf{z}$ to the clean sample. That is a strong, low-variance signal, and a biased one: many clean images can pair with the same noise vector, so the regression is pulled toward their average, the blur of the denoiser rather than the origin of one trajectory. At large $N$ the target is the model's own output one grid step lower, and the bias goes away, but the two outputs are nearly identical and the loss is dominated by noise. The schedule starts small and grows:

On CIFAR-10, $s_0 = 2$, $s_1 = 150$, $\mu_0 = 0.9$ and $K = 800\text{k}$ iterations, so $N$ goes 2, 54, 76, 107, 151 at 0, 100k, 200k, 400k and 800k iterations, and $\mu$ goes 0.9, 0.9961, 0.9972, 0.9980, 0.9986. The EMA rate is tied to $N$ because a finer grid needs a steadier target: when the two ends of the loss are almost the same point, a target that tracks every online update turns the loss into a comparison of two copies of the same noise, and a slow average smooths it. ImageNet uses $s_1 = 200$ and $\mu_0 = 0.95$.

The training step, with the two schedules in place:

# one consistency-training step: no teacher, no solver
N  = N_schedule(k)                    # 2 at k = 0, 151 at k = 800k
mu = mu_schedule(k)                   # 0.9 at k = 0, 0.9986 at the end
x  = batch(); z = randn_like(x)
n  = randint(1, N - 1)
loss = lpips(f_online(x + t[n+1] * z, t[n+1]),
             f_target(x + t[n] * z, t[n]).detach())    # (13)
loss.backward(); opt.step()
theta_target = mu * theta_target + (1 - mu) * theta

Fixing $N$ instead does worse in both directions, in the paper's Figure 3d: a small fixed $N$ converges fast to a poor FID, a large fixed $N$ converges slowly, and the schedule reaches a better FID than either. The model is initialized randomly, not from a diffusion model, and never sees one; the paper also derives continuous-time versions of both losses (Appendix B), which drop the grid entirely but need Jacobian-vector products through forward-mode autodiff, and the experiments stay discrete.

What the numbers say

The paper's main distillation baseline is progressive distillation (PD), the only earlier method that also avoids generating a synthetic dataset. PD halves the sampler's step count in stages, training each student to match two steps of its teacher. Figure 6 has the FID (Fréchet Inception Distance, lower is better) for one and two network evaluations on all four datasets.

On CIFAR-10 a single consistency-distillation call scores 3.55 against PD's 8.34, and two calls score 2.93 against 5.58. For scale, DDIM with 10 steps scores 8.23, the DDPM sampler with 1000 steps 3.17, and StyleGAN2-ADA 2.92: the two-step distilled model ties the GAN and the one-step model beats a 10-step ODE sampler. On ImageNet 64×64 the one- and two-step FIDs are 6.20 and 4.70 against PD's 15.39 and 8.95, with precision and recall (0.68, 0.63) both above PD's. On LSUN Bedroom the numbers are 7.80 and 5.22 against 16.92 and 8.47, on LSUN Cat 11.0 and 8.84 against 29.6 and 15.5. The one exception the paper reports is one-step Bedroom with the squared $\ell_2$ loss, where CD comes in slightly under PD; with LPIPS it wins there too. The teacher stays ahead everywhere: 2.04 on CIFAR-10 with 35 calls, 2.44 on ImageNet with 79.

Consistency training, with no teacher at all, reaches 8.70 one-step and 5.83 two-step on CIFAR-10. That is level with PD's distilled one-step model, and far ahead of every non-adversarial single-step generator in the table (the best flow or VAE, DC-VAE, is at 17.9), while behind the GANs (StyleGAN-XL 1.85). On ImageNet it is 13.0 and 11.1, on Bedroom 16.0 and 7.85, on Cat 20.7 and 11.7, with lower recall than the distilled models (0.47 against 0.63 one-step on ImageNet), so it covers fewer modes. The paper's Figure 5 puts EDM, one-step CT and two-step CT samples side by side from the same noise vectors: the layouts match even though the two models were trained independently, which the authors read as evidence that CT is not collapsing onto a subset of the data.

The setup behind those numbers: class-conditional models on ImageNet and unconditional elsewhere, batch 512 on CIFAR-10 and 2048 on the larger sets, 800k iterations on CIFAR-10 and 600k to 1000k on the larger sets, on 8 and 64 A100s respectively, Rectified Adam with no weight decay, and a second EMA of the online weights (rate 0.9999 on CIFAR-10) for the evaluated model, as in EDM.

Limits and what came next

A distilled model cannot beat its teacher's trajectories, since (10) bounds its distance from the teacher's consistency function and nothing else, so it trails EDM (FID 3.55 at one call against 2.04 at 35 calls on CIFAR-10). The tables report one and two steps only (the paper's Figure 4 extends the curves further), and the levels $\tau_n$ have to be searched per model and dataset.

The from-scratch recipe has a flaw the authors found themselves. In Improved Techniques for Training Consistency Models (Song and Dhariwal, 2023), Proposition 1 shows that when the target weights $\theta^-$ differ from $\theta$, the discrete loss (13) converges as $N\to\infty$ to a limit that depends only on the difference between the two weight sets and not on the data. Setting $\mu = 0$ removes the problem: the target becomes the online network with the gradient blocked, which is also the condition the paper's own continuous-time Theorem 6 already required. With that change, a pseudo-Huber loss in place of LPIPS, a weighting $\lambda(t_n) = 1/(t_{n+1} - t_n)$, a lognormal distribution over $n$ instead of uniform, and a schedule that doubles $N$ at fixed intervals instead of (14), ending at 1281 rather than 151, consistency training reaches 2.51 one-step and 2.24 two-step FID on CIFAR-10 and 3.25 and 2.77 on ImageNet 64×64, past the distilled numbers here and past StyleGAN2-ADA, without a teacher.

The distillation route scaled first. Latent Consistency Models (Luo et al., 2023) applied consistency distillation to Stable Diffusion's latent space, producing 768×768 images in two to four steps after 32 A100-hours of distillation. The continuous-time losses from Appendix B, made trainable at scale by sCM (Lu and Song, 2024), are the base of rCM, which adds a score-distillation term to keep the few-step student's diversity.