EDM: Elucidating the Design Space of Diffusion-Based Generative Models

A diffusion model is a short list of independent choices, and each one can be picked well on its own.

This paper, known as EDM, rewrites DDPM, score SDEs and DDIM in one framework: a denoiser, an ODE whose time variable is the noise level, and a table of separate choices for the noise schedule, the step spacing, the solver, the network scaling and the loss. Picking each choice well reaches FID 1.97 on CIFAR-10 in 35 network evaluations.

Take a diffusion model somebody else trained, swap in a different sampler, and get the same image quality in 35 network calls instead of 256. Then retrain with a different loss and beat the state of the art.

By mid-2022 there were three well-known ways to build a diffusion model, and each came as a package. DDPM defined a Markov chain over 1000 discrete steps and derived its loss from a variational bound. Song et al.'s score SDE work replaced the chain with a stochastic differential equation and trained a network to output the score, the gradient of the log density. DDIM kept DDPM's network and found a deterministic sampler for it. Each paper fixed its noise schedule, its time discretisation, its network input and output scaling, and its loss weighting together, and derived them from the same underlying theory, so a reader could not tell which choices mattered and which were incidental.

Karras, Aittala, Aila and Laine at NVIDIA set out to separate the choices. The paper does three things. It rewrites all three families as one denoiser plus one ordinary differential equation, with every remaining decision listed in a single table. It then improves sampling, on models it did not train: a better time discretisation, a second-order solver, a better noise schedule, and a tuned amount of injected noise, which together cut the network evaluations needed for a given FID by 7.3× on the VP (variance-preserving) model and 300× on the VE (variance-exploding) model, the two pre-trained score SDE networks, and 3.2× on DDIM. And it improves training: a principled scaling of the network's inputs, outputs and skip connection, a loss weighting that makes every noise level start at loss 1, a log-normal distribution over which noise levels to train on, and image augmentation that does not leak into the generated images. The retrained models reach FID 1.79 on class-conditional CIFAR-10, 1.97 unconditional, and 1.36 on ImageNet 64×64, with 35 network evaluations per CIFAR-10 image.

FID, the Fréchet inception distance, compares the mean and covariance of generated and real images in the feature space of an Inception classifier, computed here on 50,000 generated images; lower is better. NFE, the number of function evaluations, counts how many times the denoiser network runs per generated image, which is what sampling time is proportional to. The rest of this page follows the paper's order: the framework, then sampling, then training.

One denoiser, one ODE

Start with a data distribution $p_{\text{data}}(\mathbf{x})$ of images scaled to $[-1, 1]$, with per-pixel standard deviation $\sigma_{\text{data}}$, which the paper takes to be 0.5. Adding Gaussian noise of standard deviation $\sigma$ to every image gives a blurred distribution $p(\mathbf{x}; \sigma)$. For $\sigma = 0.002$ it is indistinguishable from the data; for $\sigma = 80$ it is indistinguishable from pure noise, because a signal of amplitude 0.5 buried under noise of amplitude 80 contributes nothing an observer could recover. Generation runs from noise level 80 down to 0: draw $\mathbf{x}_0 \sim \mathcal{N}(\mathbf{0}, 80^2 \mathbf{I})$, and move it through a sequence of noise levels $\sigma_0 = 80 > \sigma_1 > \dots > \sigma_N = 0$ so that at each stage the sample is distributed as $p(\mathbf{x}; \sigma_i)$. Whatever arrives at $\sigma_N = 0$ is a sample from the data.

The tool for moving between noise levels is a denoiser. Define $D(\mathbf{x}; \sigma)$ as the function that minimises the squared error of reconstructing a clean image from its noisy version, at every noise level separately:

The minimiser of a squared error is a conditional mean, so $D(\mathbf{x}; \sigma)$ is the average of every clean image that could have produced $\mathbf{x}$, weighted by how likely each was to do so. For a finite dataset of $Y$ images that average has a closed form (paper Appendix B.3):

Take a one-dimensional dataset of three points, $\{-1.25, -0.35, 1.05\}$, and ask what the ideal denoiser returns at $x = 0.2$ with $\sigma = 0.5$. The three Gaussian weights are 0.0149, 0.5461 and 0.2357, so the point at −1.25 contributes almost nothing and $D = 0.0474$, a value between the two nearest data points, closer to the nearer one. At $\sigma = 80$ the three weights are nearly equal for any $x$ in the plausible range, and $D$ returns the dataset mean, −0.183, whatever the input. An ideal denoiser has three regimes: at low noise it snaps to the nearest data point, at high noise it returns the mean, and in between it returns a blend.

Differentiating the log of the blurred density gives the score, and it comes out in terms of the same denoiser:

In the worked example the score at $x = 0.2$ is $(0.0474 - 0.2)/0.25 = -0.61$: it points toward the two data points on the left. Identity (3) is why nobody needs the normalising constant of $p(\mathbf{x}; \sigma)$: a network trained by plain regression on (2) yields the score by a subtraction and a division. The NCSN line of work trained for the score directly; EDM treats every score network, including those, as a denoiser, and everything from here on is written in terms of $D$.

To move a sample between noise levels without changing which distribution it follows, the paper uses the probability-flow ODE from the score SDE paper. Choose a schedule $\sigma(t)$ that says what the noise level is at time $t$. Then the ODE

has the property that a sample of $p(\mathbf{x}; \sigma(t_a))$ evolved from $t_a$ to $t_b$, forward or backward, is a sample of $p(\mathbf{x}; \sigma(t_b))$. Forward in time the noise level rises and the equation pushes the sample away from the data along the score; backward it pulls the sample toward the data. Substituting (3) and choosing the schedule $\sigma(t) = t$ (the time variable is the noise level itself) turns (1) into

Read the right-hand side as a direction: from the current point, aim away from the denoiser's guess of the clean image, scaled by one over the current noise level. Going backward in time reverses it: aim at the guess. In the worked example, at $x = 0.2$ and $t = 0.5$ the velocity is $(0.2 - 0.0474)/0.5 = 0.305$, and one Euler step backward from $t = 0.5$ to $t = 0.3$ moves the point to $0.2 - 0.2 \times 0.305 = 0.139$. Generation is this step repeated from $t = 80$ to $t = 0$, with a neural network $D_\theta$ standing in for the ideal $D$.

Some families also scale the signal over time. DDPM-style models keep the total variance near one by shrinking the image as noise is added, so their state is $\mathbf{x} = s(t)\, \hat{\mathbf{x}}$ for a scale function $s(t)$ and an unscaled variable $\hat{\mathbf{x}}$. The general ODE that covers this case is

and the paper's Appendix B.2 shows that every probability-flow ODE in the literature is (4) for some pair $\sigma(t), s(t)$: changing $\sigma(t)$ reparameterises time, changing $s(t)$ reparameterises the state, and neither changes the path a sample takes through the space of noise levels. The table below lists the remaining choices as rows. Each column is a published method, and the rows can be mixed across columns.

The VP constants are $\beta_d = 19.9$ and $\beta_{\min} = 0.1$, which put its noise level at $t = 1$ at 152 and its scale at 0.0066, so the scaled state has unit variance there. VE runs $\sigma$ from 0.02 to 100; EDM from 0.002 to 80. The four $c$ functions in the middle block describe how each method wraps its raw network, and they are the subject of the preconditioning section below.

Time is the noise level

A numerical ODE solver replaces the curve $\mathbf{x}(t)$ with straight segments, and the error of each segment grows with how much the curve bends over it. So a schedule that makes the solution paths straight in the $(t, \mathbf{x})$ plane is a schedule that lets the solver take fewer, longer steps. The paper argues that $\sigma(t) = t$ with $s(t) = 1$ is that schedule, for the following reason. With it, a single Euler step from any $(\mathbf{x}, t)$ all the way to $t = 0$ lands at $\mathbf{x} - t \cdot (\mathbf{x} - D)/t = D(\mathbf{x}; t)$, the denoiser's output. The tangent of every solution path points at the current denoised image. At high noise that image is close to the dataset mean and barely changes as the noise level drops; at low noise it is close to the nearest data point and again barely changes. Only in a middle band, where the denoiser is switching from "the mean" to "this particular image", does the tangent swing, and there the path curves.

Figure 1 draws the exact solution paths for the three-point dataset in the time coordinate of each schedule. With $\sigma(t) = t$ the paths are straight lines fanning out from the three data points, bending only in a band around $t \approx 0.5$ where the denoiser switches between points. Switch to the VE schedule, $\sigma(t) = \sqrt{t}$, and the same paths, plotted against $t = \sigma^2$, become parabolic arcs. Switch to VP and the $s(t)$ scaling squeezes the fan toward zero at high noise and bends every path. Hovering the plot draws the tangent at that point extended to $t = 0$, which is where a single Euler step from there would land, next to the denoiser's output at the same point.

Under VE the single Euler step lands only halfway to $D$, because in $t = \sigma^2$ the slope at the start of the step is half the average slope across it; under VP the scaling term adds a further bend. On real models the effect is the teal curves of Figure 4 below (red in the paper's Fig. 2): keep the pre-trained VP network and change only the schedule the sampler integrates in, and FID goes from 2.88 at 255 evaluations to 2.93 at 35. DDIM already uses $\sigma(t) = t$, so its curve does not move.

Where to put the steps

The solver also needs the discrete noise levels $\sigma_0 > \sigma_1 > \dots > \sigma_{N-1}$ at which it evaluates the network, and Appendix D.1 measures where the error comes from. With uniform steps in $\sigma$ of width 1.25 on the VE CIFAR-10 model, one Euler step incurs a root-mean-square error of about 0.56 per pixel value at $\sigma \le 0.5$ and far less at high noise. The steps therefore have to shrink as the noise level falls. The paper parameterises the spacing with one exponent:

with $\sigma_{\min} = 0.002$ and $\sigma_{\max} = 80$. At $\rho = 1$ the levels are evenly spaced in $\sigma$; as $\rho$ grows they crowd toward $\sigma_{\min}$, and in the limit they become the geometric sequence of the original VE sampler. For $N = 18$ and $\rho = 7$ the levels are 80, 57.6, 40.8, 28.4, 19.4, 12.9, 8.40, 5.32, 3.26, 1.92, 1.09, 0.585, 0.296, 0.140, 0.060, 0.023, 0.0075, 0.002, then 0. The first step divides the noise level by 1.39, the last by 3.8: a geometric spacing would divide by 1.87 every time, so $\rho = 7$ takes smaller multiplicative steps at the top and larger ones at the bottom than a geometric spacing would, spending more of its budget on high noise levels.

The value of $\rho$ is chosen against two different targets. Measured per-step numerical error is most even across noise levels at $\rho = 3$; with Heun's method that setting keeps the per-step RMSE within 0.0030 to 0.0045 everywhere. FID, however, keeps improving up to $\rho$ between 5 and 10, because errors at low noise damage the image more than errors at high noise, where the exact value of $\sigma_{\max}$ was arbitrary to begin with. The paper fixes $\rho = 7$ for everything else. When a pre-trained model only supports a discrete set of levels, as the 1000-step ImageNet model does, each $\sigma_i$ is rounded to the nearest level the model was trained on.

Heun: two network calls per step

Euler's method evaluates the slope at the start of a step and assumes it holds across the step, which is a local error of order $h^2$ in the step size $h$. Heun's method, also called the improved Euler method or the explicit trapezoidal rule, takes the Euler step, evaluates the slope again at the landing point, and then re-takes the step with the average of the two slopes. Its local error is of order $h^3$, at the cost of one extra network evaluation per step. The paper's Algorithm 1 is exactly this, written for a general $\sigma(t), s(t)$; with $\sigma(t) = t$ and $s(t) = 1$ it reduces to the code the authors ship:

# Algorithm 1 with sigma(t)=t, s(t)=1 (generate.py, edm_sampler, S_churn=0)
t = edm_levels(N=18, rho=7, smin=0.002, smax=80)   # 18 levels, then t[18] = 0
x = randn(B, 3, 32, 32) * t[0]                     # x_0 ~ N(0, 80^2 I)
for i in range(18):                                # 35 calls of D in total
    d  = (x - D(x, t[i])) / t[i]                   # dx/dt at t_i, Eq. (1)
    xe = x + (t[i+1] - t[i]) * d                   # Euler step to t_{i+1}
    if t[i+1] > 0:                                 # no correction at t = 0
        d2 = (xe - D(xe, t[i+1])) / t[i+1]         # slope at the landing point
        xe = x + (t[i+1] - t[i]) * (d + d2) / 2    # trapezoid: mean of slopes
    x = xe

The step to $t = 0$ cannot evaluate the slope at the landing point, since the slope divides by $t$, so the last step is a plain Euler step. That is why 18 steps cost 35 evaluations and 40 steps cost 79, and why every NFE in the paper's Heun results is odd. The worked example continues: from $x = 0.2$ at $t = 0.5$, the Euler step landed at 0.139 for $t = 0.3$. The denoiser at that landing point returns −0.299, so the second slope is $(0.139 + 0.299)/0.3 = 1.46$, five times the first slope of 0.305. Averaging the two and re-taking the step gives $0.2 - 0.2 \times 0.883 = 0.023$, a much larger move toward the point −0.35 than Euler's. The second evaluation measures how much the slope changes inside the step, which a single evaluation at the start cannot.

Figure 3 runs both solvers on a two-dimensional dataset of six clusters, with the same 14 starting points, the same $\rho = 7$ levels, and an exact solution for comparison. Drag $N$ and read the two error numbers. At $N = 1$ both solvers take the single step to $D(\mathbf{x}_0; \sigma_{\max})$ and every path collapses onto the centre of the data. For $N$ up to 5 Heun is no better than Euler at the same $N$, and at $N = 3$ one of its paths shoots off the canvas: the first step lands near the centre, the second slope is evaluated there at $\sigma = 0.40$ where the denoiser already points hard at one cluster, and averaging in that slope over a step of width 7.6 overshoots. From $N = 6$ Heun at $N$ steps matches Euler at $2N$ steps, the fair comparison at equal NFE, and from there the gap opens: at $N = 12$ Heun's endpoint error is 0.046 against 0.26 for Euler at 24 steps, and at $N = 18$ (35 NFE) it is 0.017 against 0.24.

On the three pre-trained models the paper stacks its sampling changes one at a time, and Figure 4 shows the resulting FID against NFE. The original samplers (grey) are re-implemented inside the framework (violet) and come out slightly better because of small oversights in the original code and a more careful handling of DDIM's discrete levels. Adding Heun and the $\rho = 7$ levels (amber) reaches the same FID at far fewer evaluations: the VP model goes from FID 2.79 at 512 evaluations to 2.88 at 255, the VE model from 4.78 at 8192 to 4.23 at 191, and ImageNet-64 from 2.73 at 384 to 2.64 at 79. Setting $\sigma(t) = t$ (teal) then brings VP to 2.93 at 35 and VE to 3.73 at 27. The dashed adaptive RK45 solver, run with the same schedule, needs 115, 93 and 131 evaluations for comparable FID, so its adaptivity does not pay for itself here.

Appendix D.2 also tries the other two-stage second-order Runge-Kutta variants, which evaluate the second slope at a fraction $\alpha$ of the way across the step: midpoint ($\alpha = 1/2$), Ralston ($2/3$), Heun ($1$). The measured optimum was $\alpha = 1.1$, which overshoots the step and has no theoretical justification, so the authors kept $\alpha = 1$. Heun has a further practical advantage: its second evaluation sits exactly at $t_{i+1}$, a level a discretely trained network has seen, whereas the midpoint variants evaluate the network at levels in between.

Adding noise back on purpose

Deterministic sampling has one output per starting noise and can be inverted to encode a real image, but the score SDE paper and DDIM both reported that adding fresh noise during sampling gives better FID. Since the ODE and the SDE produce the same marginal distributions in theory, what does the noise do? The paper writes the general SDE as the probability-flow ODE plus a Langevin term:

with $\omega_t$ a standard Wiener process, the plus sign for forward time and the minus sign for the reverse SDE used in generation. The two Langevin terms cancel in their effect on the noise level: the decay term, by (3), equals $\beta(t)(D - \mathbf{x})\,\mathrm{d}t$, which removes a fraction of the current noise, and the injection term adds the same amount back. So $\beta(t)$ sets the rate at which existing noise is swapped for new noise, and any choice of it preserves the marginals. The score SDE paper's SDEs correspond to the particular choice $\beta(t) = \dot\sigma(t)/\sigma(t)$, for which the score term drops out of the forward SDE; Karras et al. point out that nothing makes that choice special, and that the amount of churn should be tuned empirically.

Why does churn help? The Langevin term pulls the sample toward the correct marginal at the current noise level, so it corrects errors the earlier steps made. With an exact denoiser and infinitesimal steps there would be nothing to correct. With a learned denoiser and 18 steps there is, and the correction is worth having up to the point where its own discretisation error and the denoiser's imperfections take over.

The paper's stochastic sampler, Algorithm 2, keeps the Heun step and adds the churn as a separate sub-step. At step $i$ with the sample at noise level $t_i$, first raise the noise level to $\hat t_i = t_i + \gamma_i t_i$ by adding fresh Gaussian noise of standard deviation $\sqrt{\hat t_i^2 - t_i^2}$, then take one Heun step of the ODE from $\hat t_i$ down to $t_{i+1}$. Each sub-step keeps the sample on the correct marginal (up to the ODE step's truncation error), so their alternation does too. The difference from a textbook Euler-Maruyama step is that the network is evaluated at the state after the noise was added, $(\hat{\mathbf{x}}_i, \hat t_i)$, rather than at the state before it. In the limit of tiny steps the two agree; at 18 steps they do not.

Three constants set the churn per step:

The clamp at $\sqrt 2 - 1 = 0.414$ means $\hat t_i \le \sqrt 2\, t_i$, and the added noise then has standard deviation at most $t_i \sqrt{2 - 1} = t_i$: a step never injects more new noise than the noise already present. The range $[S_{\text{tmin}}, S_{\text{tmax}}]$ switches churn off at very low and very high noise levels, where it was found to oversaturate colours. A fourth constant, $S_{\text{noise}}$, multiplies the injected noise; values slightly above 1 (1.003 to 1.007) restore detail that heavy churn washes out, which the authors read as the learned denoiser removing slightly too much noise, the regression toward the mean expected of any denoiser trained with a squared loss.

Figure 5 runs Algorithm 2 on the six-cluster toy. The left panel shows the noise level over the 18 steps, an amber climb followed by a teal descent at each step; the right panel shows one sample path with the same colours. For the CIFAR-10 VP model the tuned setting was $S_{\text{churn}} = 30$ at $N = 192$ steps, which gives $\gamma = 0.156$, $\hat t = 1.156\, t$, and new noise of standard deviation $0.58\, t$ at every step inside $[0.01, 1]$. For the pre-trained ImageNet-64 model it was $S_{\text{churn}} = 80$ at $N = 256$, so $\gamma = 0.31$ and new noise of $0.85\, t$ per step inside $[0.05, 50]$.

Figures 6 and 7 hold the paper's measurements. With all four constants tuned by grid search per model (the table in Appendix E.2), stochastic sampling brings the VP model from FID 2.93 to 2.27, VE from 3.73 to 2.23, and the pre-trained ImageNet-64 model from 2.64 to 1.55, against the model's own published 2.07. At the time, 1.48 (cascaded diffusion), 1.55 (classifier-free guidance) and 1.52 (StyleGAN-XL) were the best reported ImageNet-64 numbers. Figure 7 shows the shape behind the tuning: FID falls with $S_{\text{churn}}$ up to a model-specific optimum and rises past it, and each of the two refinements, $S_{\text{noise}}$ and the level range, moves the whole curve down.

The paper is explicit about the cost. The four constants had to be found by grid search separately for each model, and they encode properties of that model's errors, so a paper that evaluates architecture changes with a stochastic sampler is also, in part, evaluating how well the churn heuristics happen to suit each architecture. The section on training returns to this: once the network is trained with the recipe below, the benefit of churn shrinks and, on CIFAR-10, disappears.

Preconditioning the network

A network trained to output $D(\mathbf{x}; \sigma)$ directly would face inputs whose magnitude ranges from 0.5 at low noise to 80 at high noise, and would have to produce outputs of magnitude 0.5 from both. Every method therefore wraps a raw network $F_\theta$ in fixed scalings. The VP and DDPM families divide the input by $\sqrt{\sigma^2 + 1}$ (VE leaves it unscaled; for VP this is exactly its $s(t)$), have $F_\theta$ predict the noise scaled to unit variance, and reconstruct $D = \mathbf{x} - \sigma F_\theta$. At $\sigma = 80$ that reconstruction multiplies every error the network makes by 80, and requires the network to reproduce a noise pattern of amplitude 80 to within a fraction of the 0.5-amplitude signal it hides. Predicting the signal would be easier there, but predicting the noise is easier at low $\sigma$, where the signal is nearly the input. EDM lets the wrapper choose per noise level, through a skip connection:

Substituting (7) into the loss (2), weighted by $\lambda(\sigma)$ and averaged over noise levels drawn from a training distribution $p_{\text{train}}(\sigma)$, and collecting terms shows what $F_\theta$ is being trained to output:

Four requirements then fix the four functions, in the paper's Appendix B.6. The input to $F_\theta$ should have unit variance; since $\mathbf{y} + \mathbf{n}$ has variance $\sigma_{\text{data}}^2 + \sigma^2$, this gives $c_{\text{in}} = 1/\sqrt{\sigma^2 + \sigma_{\text{data}}^2}$. The target should have unit variance, which sets $c_{\text{out}}^2 = (1 - c_{\text{skip}})^2 \sigma_{\text{data}}^2 + c_{\text{skip}}^2 \sigma^2$, the variance of the numerator of the target. The network's errors are multiplied by $c_{\text{out}}$ on the way to $D_\theta$, so choose $c_{\text{skip}}$ to make $c_{\text{out}}$ as small as possible: setting the derivative of that quadratic to zero gives $c_{\text{skip}} = \sigma_{\text{data}}^2 / (\sigma^2 + \sigma_{\text{data}}^2)$, and substituting back, $c_{\text{out}} = \sigma \sigma_{\text{data}} / \sqrt{\sigma^2 + \sigma_{\text{data}}^2}$. The fourth, $c_{\text{noise}} = \tfrac14 \ln \sigma$, is chosen empirically; it turns a noise level between 0.002 and 80 into a conditioning scalar between −1.55 and 1.10.

The numbers at three noise levels show what the wrapper does. At $\sigma = 0.01$: $c_{\text{skip}} = 0.9996$, $c_{\text{out}} = 0.010$, and the target is $0.04\,\mathbf{y} - 100\,\mathbf{n}$, whose variance is entirely noise, so $F_\theta$ is a noise predictor and its output is scaled down by 100 before it touches the input. At $\sigma = 0.3$, the median training level: $c_{\text{skip}} = 0.735$, $c_{\text{out}} = 0.257$, $c_{\text{in}} = 1.715$, and the target is $1.03\,\mathbf{y} - 2.86\,\mathbf{n}$, 27% signal variance and 73% noise. At $\sigma = 10$: $c_{\text{skip}} = 0.0025$, $c_{\text{out}} = 0.499$, and the target is $2.0\,\mathbf{y} - 0.005\,\mathbf{n}$, 99.8% signal, so $F_\theta$ predicts the clean image scaled to unit variance and the skip contributes nothing. The crossover, where the target is half signal and half noise, is at $\sigma = \sigma_{\text{data}} = 0.5$, where $c_{\text{skip}} = 1/2$.

On its own this change leaves CIFAR-10 FID roughly where it was: config D in the table below moves it by between −0.02 and +0.27. Its effect shows on the 64×64 VE model, whose FID goes from 41.62 to 3.39 on FFHQ and from 15.04 to 3.81 on AFHQv2, and in the authors' report that training became robust enough to redesign the loss without destabilising it.

Which noise levels to train on

Equation (8) says the raw network sees an effective weight of $\lambda(\sigma) c_{\text{out}}(\sigma)^2$ per sample. The paper sets

so that every noise level carries weight 1. The output layer of $F_\theta$ is initialised to zero, so at the start of training $D_\theta = c_{\text{skip}}\mathbf{x}$ and the expected loss at every $\sigma$ is exactly 1 (Appendix B.6 works the algebra; the grey curve in Figure 9 is the measurement). With the earlier weighting $\lambda = 1/\sigma^2$ the initial loss would vary by orders of magnitude across the range and the gradient would be dominated by whichever levels happened to be loudest.

Where should training draw its noise levels from? The trained loss curves answer it. On CIFAR-10 the loss after training is still 0.94 at $\sigma = 0.001$ and 0.52 at 0.01: the noise is so small that separating it from the image is both hard and, for generation, pointless. At the other end it is 0.67 at 10, 0.92 at 30 and 0.99 at 100: the correct answer is close to the dataset mean and the target, one particular training image, is always far from it. In between it falls below 0.35 for $\sigma$ from 0.03 to 3 and bottoms out at 0.10 near $\sigma = 0.44$ (0.07 near 0.49 on FFHQ). Training can only reduce the loss in that band, and the paper concentrates the training samples there with a log-normal distribution: $\ln \sigma \sim \mathcal{N}(P_{\text{mean}}, P_{\text{std}}^2)$ with $P_{\text{mean}} = -1.2$ and $P_{\text{std}} = 1.2$. Its median is $e^{-1.2} = 0.30$, 66% of the draws land in $[0.1, 1]$, 6.7% below 0.05, and 16% above 1; a draw above 80 or below 0.002 happens about once in sixty thousand samples.

A training step draws a noise level per image from the log-normal, adds noise at that level, runs the wrapped network, weights the squared error, and back-propagates into $F_\theta$ only; the four $c$ functions are fixed formulas with no parameters:

# one EDM training step (training/loss.py, EDMLoss); CIFAR-10, batch of 4
y     = batch                              # clean images [4, 3, 32, 32] in [-1, 1]
ln_s  = -1.2 + 1.2 * randn(4)              # ln(sigma) ~ N(P_mean, P_std^2)
sigma = exp(ln_s).view(4, 1, 1, 1)         # e.g. [0.30, 0.05, 1.9, 0.11]
n     = sigma * randn_like(y)              # noise at each sample's own level
D     = precond(F, y + n, sigma)           # Eq. (7): c_skip*x + c_out*F(...)
lam   = (sigma**2 + 0.25) / (0.5 * sigma)**2   # lambda = 1 / c_out^2
loss  = (lam * (D - y)**2).mean()          # Eq. (2) weighted; grad reaches F only

For the batch in the listing the four noise levels 0.30, 0.05, 1.9 and 0.11 get weights 15.1, 404, 4.3 and 86.6, and the four targets handed to $F_\theta$ range from 99% noise variance (at 0.05) to 94% signal (at 1.9). Config E in the table is this loss on top of config D, and on CIFAR-10 it is the largest of the paper's own three training changes (configs D, E, F): CIFAR-10 unconditional VP goes from 2.29 to 2.05, VE from 3.10 to 1.99. On the 64×64 VE models config D is larger (FFHQ 41.62 to 3.39, AFHQv2 15.04 to 3.81). Concurrent work by Choi et al. changed only $\lambda$ and found a smaller gain; the paper attributes the difference to changing the weight and the sampling distribution together.

The last training change borrows from GANs. Diffusion models on small datasets overfit, so the paper applies the geometric augmentations of the ADA pipeline (adaptive discriminator augmentation, from StyleGAN2-ADA): flips, rotations, scalings and translations applied to each training image before adding noise. To keep the augmentations out of the generated images, the augmentation parameters are fed to $F_\theta$ as a nine-dimensional conditioning vector, and set to zero at sampling time, so the network learns to generate un-augmented images on request. Config F adds this and reaches the final numbers. The full ladder, evaluated with the 35-evaluation (79 at 64×64) deterministic sampler:

Configs B and C are hyperparameter changes: batch size 512 at 32×32 (256 at 64×64) instead of 128, on 8 GPUs instead of 4, learning rate 0.001 for CIFAR-10 (unchanged at 0.0002 for the 64×64 sets) with a 10-million-image warm-up, no gradient clipping, an exponential moving average of the weights with a half-life of 0.5 million images, dropout tuned per dataset, and the 4×4 resolution layers removed with the 16×16 layers doubled in width, for about 56 million parameters at 32×32. Each CIFAR-10 model trains for 200 million images, about 400,000 iterations, in roughly two days. By config F the VP and VE columns differ only in the network architecture, DDPM++ against NCSN++, and their FIDs agree to within 0.01.

What the recipe buys

The headline numbers are FID 1.79 on class-conditional CIFAR-10 and 1.97 unconditional, against the previous records of 1.85 (StyleGAN-XL) and 2.10, with 35 network evaluations and deterministic sampling. For ImageNet-64 the authors trained the unchanged ADM architecture from Dhariwal and Nichol, about 296 million parameters, with config E for two weeks on 32 GPUs at batch 4096, and reached FID 1.36 against the previous 1.48. Neither number used guidance or a separate super-resolution stage.

The two halves of the paper interact. On the pre-trained models, stochastic sampling was worth 0.7 to 1.5 FID points. On the retrained CIFAR-10 models, the best result came from the deterministic sampler and every amount of churn made FID worse; the retrained ImageNet-64 model still gained from churn, but its optimum moved from $S_{\text{churn}} = 80$ to 40 and the gain shrank. The paper's reading is that churn corrects the denoiser's errors, so a better-trained denoiser leaves less to correct, and that more diverse datasets keep benefiting for longer. The interaction between the training objective and stochastic sampling is left open.

The limitations the authors state: the churn constants are heuristics found by grid search per model; the degradation they guard against is not understood beyond the hypothesis that the learned denoiser induces a field that is not the gradient of any density (a non-conservative field); the parameter values were tuned at 32×32 and 64×64 and may need re-tuning at higher resolution, where current systems add super-resolution stages, latent spaces or much larger networks on top; and the project consumed about 250 MWh on V100s. Later systems, from rCM distillation to DiffusionBlocks, state their design in the rows of the paper's Table 1: a $\sigma(t) = t$ schedule, the $\rho$ spacing, the four $c$ functions with $\sigma_{\text{data}} = 0.5$, and a log-normal over training noise levels.