ELF: Embedded Language Flows

Generate text by flow matching in a frozen encoder's embedding space, and turn the vectors into tokens only once, at the last step.

ELF trains one transformer to denoise T5 embeddings of a whole 1,024-token passage and, at the final step, to read them out as tokens. Techniques from image diffusion carry over without change, and the 105M-parameter model reaches a GPT-2 Large perplexity of 24 in 32 sampling steps.

This page builds ELF from the flow-matching update: an x-prediction Euler step in a 2-D toy you can drive, the noise schedule, the single decoding pass, and how self-conditioning supplies the “condition” that classifier-free guidance needs when there are no labels. The results are read next to their entropy, and two places where the official code differs from the paper's pseudocode are flagged.

Discrete and continuous diffusion for text

A diffusion language model (DLM) generates a whole passage at once by starting from noise and refining every position over a number of steps, instead of writing left to right like GPT. Two families exist. Discrete DLMs corrupt the tokens themselves: MDLM replaces tokens with a [MASK] symbol and learns to unmask them, and Duo replaces them with random vocabulary items and learns to put the right ones back. Continuous DLMs map tokens to vectors, add Gaussian noise to the vectors, and learn to remove it, the way image diffusion models such as DDPM remove noise from pixels.

The continuous family includes Diffusion-LM, CDCD and DiffuSeq (2022 to 2023), but most recent progress has come from discrete models, which report the stronger results. The ELF paper attributes part of the gap to how continuous models handle the boundary between vectors and tokens. Most of them add a token-level cross-entropy loss at every denoising step, or round each intermediate state to the nearest token vector during sampling, or both; the latent-diffusion variants avoid that but train a separate decoder network. Each choice ties the continuous trajectory to the vocabulary or adds a component.

ELF keeps the trajectory continuous until the end. A frozen T5 encoder turns the training text into one vector per token; a transformer learns to denoise those vectors with flow matching, the method behind recent image models; and the same transformer, told by a mode token that it is now decoding, turns the final vectors into tokens with one cross-entropy-trained readout. Nothing is discretized at intermediate steps, and no separate decoder exists at inference.

The embedding space: frozen T5 encoder outputs

A training passage from OpenWebText is tokenized with the T5 tokenizer and cut to $L = 1024$ tokens (sequences are packed, so a passage can span document boundaries). The token ids go through the encoder of T5-small, a 35M-parameter pretrained bidirectional transformer covered in the T5 explainer, whose output is one 512-dimensional vector per token. The code divides every coordinate by 0.2, its estimate of the standard deviation of the encoder outputs, so the clean data $x$ is a $1024 \times 512$ array with roughly unit-variance entries.

These vectors are contextual: the vector for “bank” depends on the whole passage, because every encoder layer attends over all 1,024 positions. The encoder is never trained, and for unconditional generation it runs only during training, where it defines the target space. The paper compared five choices of space (Figure 5a). Pretrained contextual T5 outputs gave the best trade-off between quality and diversity, a T5-style encoder trained from scratch on OpenWebText came close behind, and the non-contextual options (T5's token embedding table, a frozen random Gaussian table, a table learned jointly with the denoiser) were worse, with the learned table worst.

How ELF trains the denoiser: flow matching with x-prediction

Flow matching defines a path from noise to data. Draw a clean array $x$, a Gaussian noise array $\epsilon$ of the same shape, and a time $t \in [0, 1]$, and form the straight-line mixture

At $t = 0$ the array is pure noise and at $t = 1$ it is the data. The velocity $v$ is constant along each straight path. A network that predicts, at every point $z_t$, the expected velocity over all noise-data pairs whose paths pass through it defines an ordinary differential equation (ODE), and integrating it from fresh noise at $t = 0$ to $t = 1$ produces new samples. ELF's code draws $\epsilon$ with standard deviation 2 instead of 1; the paper mentions this as a noise scale of 2 in its appendix. All numbers on this page use the code's scale.

Standard flow matching trains the network to output $v$ directly. ELF follows JiT (Li and He, 2025) and has the network output a guess of the clean data, $x_\theta(z_t, t)$, then converts it to a velocity. Solving $z_t = t x + (1-t)\epsilon$ for $\epsilon$ and substituting gives $v = (x - z_t)/(1 - t)$, so the predicted velocity is $v_\theta = (x_\theta - z_t)/(1 - t)$. The loss compares velocities:

The second form follows because $z_t$ cancels in $v_\theta - v$, leaving $(x_\theta - x)/(1 - t)$. The network is an x-predictor trained with a weighted x-loss whose weight grows near the data end. The code clips $1 - t$ at 0.05, which caps the weight at 400.

Take one training passage at the median time the schedule draws, $t = 0.18$. Each coordinate of $z_t$ is 0.18 times a unit-scale clean value plus noise with standard deviation $0.82 \times 2 = 1.64$, nine times larger. The network sees this $1024 \times 512$ array, outputs a $1024 \times 512$ guess $x_\theta$, and the loss weight is $1/0.82^2 = 1.49$.

Why predict $x$ rather than $v$? JiT's argument is geometric. Clean data occupies a small, structured subset of the space (for embeddings: the vectors an encoder produces for real text), while noise and velocity fill all 512 dimensions per token. A network that outputs a point near that subset has less to represent than one that must reproduce full-dimensional noise. ELF's Appendix D.1 tests this with T5-small, T5-base and T5-large encoders (512, 768 and 1,024 dimensions): x-prediction stays usable in all three, v-prediction is competitive at 512 and degrades above it, and noise prediction collapses at every size. The second reason is specific to ELF and comes back in the decoding section: at $t = 1$ the network must output clean embeddings for the token readout, and an x-predictor already does.

Following JiT, the network also squeezes its input through a bottleneck: a linear map from 512 to 128 dimensions per token, then up to the 768-wide hidden size. Of the bottlenecks the authors tried (32, 128, 512), 128 gave the best balance; 32 reached lower perplexity at lower diversity and 512 the reverse.

Sampling: each Euler step moves part of the way to the guess

Generation starts from $z_0$, a $1024 \times 512$ array of noise, and integrates the ODE with Euler steps on a grid of times $0 = t_0 < t_1 < \dots < t_N = 1$. One step from $t$ to $t'$ is $z_{t'} = z_t + (t' - t)\,v_\theta$. Substituting the x-prediction form of $v_\theta$ and collecting terms:

where $\hat{x} = x_\theta(z_t, t)$ is the network's current guess. The new point lies on the straight segment from $z_t$ to the guess, a fraction $(t' - t)/(1 - t)$ of the way along. With four uniform steps, the step from 0.25 to 0.5 goes $0.25/0.75 = 1/3$ of the way to the guess, and the last step, from 0.75 to 1, goes all the way: $z_1 = \hat{x}$. The final Euler step of any grid that ends at 1 returns the network's last guess.

At $t = 0$ the input contains no information about the data, so the best possible guess is the average of all clean data, the same point for every starting noise. A one-step sampler therefore sends every sample to that average, which is not itself a plausible embedding. More steps let the guess sharpen as the signal in $z_t$ grows. Figure 1 runs this with an exact denoiser in two dimensions, so every error on screen comes from the step size.

At N = 1 all 44 samples end on the cross. At N = 2 the second step starts at $t = 0.5$, where the noise standard deviation is 1.0 and the gap between neighbouring clusters, scaled by 0.5, is only 0.68 to 0.76, so each guess is still a blend of several clusters and none of the 44 samples ends inside one. With 4 steps 19 do, with 8 steps 40, with 16 steps 43. Step through a run and the guesses of the early steps all sit near the cross, then spread toward individual clusters as $t$ grows.

ELF does not space its steps uniformly. It draws training times from a logit-normal distribution, $t = \sigma(u)$ with $u \sim \mathcal{N}(-1.5, 0.8^2)$ and $\sigma$ the sigmoid, which puts the median at 0.18; at sampling time the code draws $N - 1$ times from the same distribution, sorts them, and adds 0 and 1. Figure 2 plots both training distributions and the resulting grid.

Under this schedule 99.7% of training draws have $t < 2/3$, where the noise standard deviation exceeds the signal coefficient, and 3% have $t > 0.5$. The sampling grid follows the same shape: with N = 32 the largest interior time averages 0.53 and ranges from about 0.35 to 0.75 between draws, so most steps are spent below $t = 0.3$ and the final step, which returns the guess, spans roughly the second half of the time axis. The network commits to its answer from a state that is still about two parts noise to one part signal per coordinate. A likely reason this works, which the paper does not test directly, is that each guess pools evidence over 1,024 positions of 512 coordinates each, against the two coordinates of Figure 1. The paper's Appendix D.6 reports that this grid gives lower perplexity than a uniform one at every step count it tried, with the largest gain at few steps.

Decoding with the same network

After the last Euler step, $z_1$ is a $1024 \times 512$ array that should look like T5 encoder output for some passage. ELF needs token ids. A latent diffusion model would train a separate decoder for this; ELF reuses the denoiser. The final time step is treated as a decoding step: the network is called once more at $t = 1$ with a mode input set to “decode”, and its output goes through a projection to 32,100 logits per position, one per T5 vocabulary entry. The argmax at each position is the generated token.

Training the decode mode needs inputs. At $t = 1$ the flow-matching input equals the clean data, so training on it would teach a lookup that never sees an imperfect embedding. ELF corrupts the clean embeddings with a separate schedule, per token:

and trains with a per-token cross-entropy loss against the true tokens $s$:

Every token gets its own $p$, so one passage mixes nearly clean tokens with heavily corrupted ones. At the median $p = 0.69$, a token's vector is 0.69 times its clean value plus noise of standard deviation $0.31 \times 5 = 1.55$; about 16% of tokens get $p < 0.5$. A heavily corrupted token can be read only from its neighbours, which the transformer attends to, so the decode mode learns to use context to clean up embeddings. The paper corrupts the inputs because the embeddings the sampler hands over at $t = 1$ are never exact.

Eq. (2) writes the readout as a matrix $W$ applied to the predicted clean embedding. In the released code the logits instead come from the transformer's last hidden state, through a small two-layer head (768 to 512 with a GELU, then 512 to 32,100), in parallel with the layer that outputs $x_\theta$. The shared part is the 12-layer transformer; the Provenance panel records the difference.

Each training step uses one of the two branches: the denoising loss (1) with probability 0.8, the decoding loss (2) with probability 0.2. In the code the choice is one random draw per device per step, so at any step about a fifth of the devices train the decoder while the rest train the denoiser. The paper tested other splits (Appendix D.3); giving the denoiser less than 0.8 hurt, most under the SDE sampler. It also trained the alternative with a separate decoder: first a T5-sized decoder learns to reconstruct tokens from masked and noisy T5 embeddings, then a separate denoiser is trained against the frozen pair. The two reached a similar trade-off, with the shared network reaching lower perplexity at its low-diversity end (Figure 5b) and needing one training stage instead of two.

Shared weights also favour x-prediction: in decode mode the network must output a clean embedding, and in denoise mode an x-predicting network already outputs one, so both losses train the shared weights to output the same kind of quantity (a clean embedding). A v-predicting network would have to output a velocity in one mode and a clean embedding in the other; the paper reports that v-prediction works poorly with shared weights.

Self-conditioning, and guidance without labels

Two more pieces come from earlier diffusion work. Self-conditioning (Chen, Zhang and Hinton, 2023) gives the network its own previous guess as an extra input. During training, half the examples get a zero array in that slot. For the other half, the network first runs once with the zero input, its output $\hat{x}'$ is detached from the gradient, and the network runs again with $[z_t, \hat{x}']$ concatenated along the feature axis (1,024 features, projected back to 512 by a linear layer). Only the second pass is trained. At sampling time the previous step's guess fills the slot, so self-conditioning costs no extra passes.

Classifier-free guidance (CFG) trades diversity for quality in conditional image models by extrapolating from an unconditional prediction past a conditional one. In the flow-matching form ELF uses,

where $c$ is the condition, $\varnothing$ its absence, and $\omega$ the scale. At $\omega = 1$ this is the conditional prediction; at $\omega = 3$ it is $3v(z_t|c) - 2v(z_t|\varnothing)$, twice the conditional-minus-unconditional difference beyond the conditional prediction. (Ho and Salimans write the scale as $w = \omega - 1$.) Unconditional text generation has no class label to condition on. ELF uses the self-conditioning input as the condition: the “conditional” prediction sees the previous guess, the “unconditional” one sees zeros. Guidance with $\omega > 1$ pushes each step further in the direction that the network's own earlier guess moves it, which the paper finds lowers perplexity and entropy together.

Standard CFG costs two network passes per step. ELF uses training-time guidance instead, following improved MeanFlow (Geng et al., 2025): the guidance scale $\omega$ becomes an input of the network, and the network is trained to output the guided velocity directly. The regression target for the self-conditioned examples is

with the two network terms evaluated with gradients stopped. The examples without self-conditioning keep the plain target $x - \epsilon$, so the network's zero-input prediction stays the unguided $v(z_t|\varnothing)$ at every $\omega$. To see why (5) gives the guided field: regressing onto $x - \epsilon$ alone makes the output converge to its average over all training pairs that match the inputs, which is $v(z_t|c)$. If the network already outputs the guided field, its two outputs in (5) differ by $\omega\,(v(z_t|c) - v(z_t|\varnothing))$, so the target averages to $v(z_t|c) + (\omega - 1)(v(z_t|c) - v(z_t|\varnothing))$, which is $v_{\text{cfg}}$: the guided field is a fixed point of training. At $\omega = 1$ the correction term vanishes.

Each training example draws its own $\omega$ from [0.5, 5.0], log-uniformly in $1 + \omega$, so the median is 2 and about 21% of examples get $\omega < 1$, which moves the prediction toward the unconditional one. At inference the scale is a free input: the system-level runs use $\omega = 3$, and changing it needs no retraining and no second pass.

During training, $\hat{x}'$ is computed from $z_t$ alone, so it carries no information about $x$ that $z_t$ lacks, and a network that computed the exact posterior would give $v(z_t|c) = v(z_t|\varnothing)$; the guidance term would then be zero. For a real network the one-pass and two-pass predictions differ, and guidance scales up that difference. The paper does not analyse what the difference represents; its evidence that the scale helps is the measured trade-off in Figure 3 below.

The time, the guidance scale and the mode reach the network as extra tokens. Four learned time tokens carry the embedding of $t$, four carry the embedding of $\omega$, and four mode tokens are switched on only in decode mode, so ELF-B's transformer reads a sequence of $12 + 1024 = 1036$ vectors of width 768 and drops the first 12 at the output. This replaces the adaLN-Zero conditioning layers of the DiT architecture it otherwise follows, and cuts ELF-B from 148M to 105M parameters with a small improvement in samples (Appendix D.4). The blocks use SwiGLU, RMSNorm, rotary position embeddings and query-key normalization.

The SDE-style sampler

The deterministic sampler follows one trajectory per starting noise, and an error made early is carried to the end. Stochastic samplers for diffusion models re-inject a little noise at each step so later steps can correct it. ELF adds a simple version. At each step from $t$ to $t'$, with a scale $\gamma$, the code first steps backward toward noise:

with fresh noise $e$, and then takes an ordinary Euler step from $z_b$ at $t_b$ to $t'$ using the guess $x_\theta(z_b, t_b)$. With $\gamma = 0$ it is the ODE sampler. Take a 32-step run at $t = 0.10$, $t' = 0.12$, $\gamma = 1.5$: then $\alpha = 0.97$ and $t_b = 0.097$, so the state gives up 3% of itself to fresh noise and the step covers 0.023 of time instead of 0.02. The signal coefficient of $z_b$ is exactly $\alpha t = t_b$, but its noise is a little below what the path formula expects at $t_b$: for a state with the exact noise level $2(1 - 0.10) = 1.80$, the backed-off state has standard deviation $\sqrt{(0.97 \cdot 1.80)^2 + (0.03 \cdot 2)^2} = 1.747$ where $2(1 - 0.097) = 1.806$ would be consistent. The paper calls the sampler “SDE-inspired” for this reason. The code always takes the last step, the one that lands on the guess, as a plain Euler step.

Algorithm 6 in the paper writes a different update: it evaluates the network at the noised state but then moves the original, un-noised $z_t$. That version never puts the fresh noise into the trajectory. The Provenance panel records the mismatch; this page follows the code. In the experiments, the SDE sampler reaches much lower perplexity than the ODE sampler at few steps (Figure 5c) at slightly lower entropy, and $\gamma$ trades the two against each other (Appendix D.6). The paper gives error correction as its hypothesis for the gain.

The training step and the sampler in code

The listing below is one ELF-B training example, condensed from the released JAX code with this page's names; every constant is the OpenWebText configuration. Only the last network call of each branch receives a gradient. The self-conditioning pass and the two passes that build the guided target run with gradients stopped, so in this condensed form a self-conditioned denoising example costs three forward passes and one backward pass.

# One ELF-B training example (unconditional, OpenWebText).
# s: 1024 token ids. Shapes per example; the batch adds a leading 512.
x = t5_small_encoder(s) / 0.2          # (1024, 512), frozen, never trained
if rand() < 0.8:                       # denoising branch, 80% of steps
    t = sigmoid(0.8 * randn() - 1.5)   # one t per sequence, median 0.18
    e = 2.0 * randn_like(x)            # noise scale 2
    z = t * x + (1 - t) * e
    v = (x - z) / max(1 - t, 0.05)     # target velocity, = x - e
    w = sample_cfg_scale(0.5, 5.0)     # guidance scale fed as an input
    x0 = stopgrad(net(cat(z, 0), t, w, mode="denoise"))   # no self-cond
    if rand() < 0.5:                   # self-conditioned example
        xs = stopgrad(net(cat(z, x0), t, w, mode="denoise"))
        v_c = (xs - z) / max(1 - t, 0.05)
        v_u = (x0 - z) / max(1 - t, 0.05)
        target = v + (1 - 1 / w) * (v_c - v_u)       # Eq. (5)
        x_pred = net(cat(z, x0), t, w, mode="denoise")
    else:
        target = v
        x_pred = net(cat(z, 0), t, w, mode="denoise")
    loss = mean(((x_pred - z) / max(1 - t, 0.05) - target) ** 2)
else:                                  # decoding branch, 20% of steps
    p = sigmoid(0.8 * randn(1024, 1) + 0.8)    # one p per token
    zt = p * x + (1 - p) * 5.0 * randn_like(x)
    logits = net(cat(zt, 0), t=1, w, mode="decode")  # (1024, 32100)
    loss = cross_entropy(logits, s)
loss.backward()                        # every branch updates the same net

Sampling for the headline 32-step result, with $\gamma = 1.5$ and $\omega = 3$, is 32 denoising passes and one decoding pass:

# Sampling (generation_utils.py / sampling_utils.py), N = 32, SDE.
ts = [0] + sorted(sigmoid(0.8 * randn(N - 1) - 1.5)) + [1]
z = 2.0 * randn(1024, 512)
x_prev = zeros(1024, 512)              # self-conditioning input
for t, t2 in pairs(ts[:-1]):           # first N - 1 intervals
    a = clip(1 - gamma * (t2 - t), 0, 1)      # gamma = 1.5 at 32 steps
    tb = a * t                                # step back toward noise
    zb = a * z + (1 - a) * 2.0 * randn_like(z)
    x_prev = net(cat(zb, x_prev), tb, w=3, mode="denoise")
    z = zb + (t2 - tb) * (x_prev - zb) / (1 - tb)
t = ts[-2]                             # last interval: plain Euler to t = 1
x_prev = net(cat(z, x_prev), t, w=3, mode="denoise")
z = z + (1 - t) * (x_prev - z) / (1 - t)      # = x_prev
tokens = argmax(net(cat(z, 0), t=1, w=3, mode="decode"))

ELF-B trains with the Muon optimizer at a learning rate of 0.002 (AdamW, the subject of the decoupled weight decay explainer, did worse in Appendix D.5 at its own tuned rate of 1e-4), batch size 512, no weight decay, a constant learning rate after half an epoch of warmup, and an exponential moving average of the weights with decay 0.9999 that is used for sampling. Five epochs of OpenWebText are about 95K steps and took 1.5 hours per epoch on 64 TPU v5p chips.

What the experiments show

Unconditional results follow the protocol of MDLM and Duo: generate 1,000 samples of 1,024 tokens, score each with GPT-2 Large, and report the generative perplexity (Gen. PPL, the exponentiated average negative log-likelihood GPT-2 Large assigns; lower means more predictable to GPT-2) together with the average unigram entropy of each sample (higher means a more varied mix of tokens). The two have to be read together. Repetitive text scores a low Gen. PPL, so a sampler can lower its perplexity by lowering its entropy; the paper's appendix figures shade entropy below 5.0 as degenerate, and the Duo paper puts samples from an autoregressive model at about 5.6.

The guidance scale $\omega$ moves a model along that trade-off. Figure 3 plots the paper's Table 10: three model sizes, both samplers, 64 steps, $\omega$ from 0.5 to 4.

Raising $\omega$ from 0.5 to 3 moves every curve down and to the left: ELF-B's SDE samples go from 36.77 at entropy 5.28 to 19.72 at 5.10. Above 3 the B and M models reverse (ELF-B at $\omega = 3.5$ jumps back to 37.56), and only ELF-L keeps improving, to 21.37 at $\omega = 4$. Larger models sit to the right at a given perplexity: at $\omega = 3$ ELF-L's SDE samples have entropy 5.28 against ELF-B's 5.10, with Gen. PPL 23.31 against 19.72. At the same $\omega$, SDE samples have lower perplexity and lower entropy than ODE samples, and at matched entropy the SDE curve is lower: ELF-B's SDE samples at entropy 5.14 score 22.53, its ODE samples at 5.15 score 26.62. In the ODE rows at $\omega = 0.5$, where entropy reaches 5.51 to 5.52, the perplexities are 68 to 104.

The system-level comparison uses ELF-B with the SDE sampler and $\omega = 3$. Averaged over six evaluation seeds it reaches 67.32 at 8 steps, 33.66 at 16 and 24.08 at 32 (standard error 0.16), at entropy 5.14 to 5.16. The baselines (MDLM, Duo, and the concurrent continuous models FLM and LangFlow, all around 170M parameters) are plotted in the paper's Figure 7a without a table. For reference, the Duo paper reports undistilled 32-step values of 96.19 for Duo at entropy 5.57 and 196.79 for MDLM at 5.75. ELF-B's lowest perplexity comes with an entropy about 0.4 below those, and its own 64-step ODE samples at entropy 5.51 score 104.29, so at matched diversity the margin over Duo is not established by the reported numbers.

Distilled baselines are a second comparison. MDLM+SDTT, Duo+DCD and FMLM add a distillation stage to sample in fewer steps. The authors distill ELF too (Appendix B): five rounds of progressive distillation (Salimans and Ho, 2022) take a 64-step teacher to a 16-step student, then 8, 4, 2 and 1. In each round the student regresses onto

where $z_r$ is where the teacher's substeps take $z_t$ by time $r$. By the Euler formula above, $\tilde{x}$ is the guess that makes one student step from $t$ land exactly at $z_r$. Figure 4 has the few-step numbers.

At 8 steps, undistilled ELF-B (67.32) is below MDLM+SDTT (112.66), Duo+DCD (118.21) and FMLM (86.50). ELF+PD is lowest at every budget: 136.10 at one step against FMLM's 168.30, and 23.18 at 8 steps. The entropy column moves the other way. ELF+PD's entropy falls from 5.26 at one step to 5.04 at 32, just above the 5.0 degenerate line, while the discrete baselines stay between 5.34 and 5.43 from 2 steps up.

The baselines train for 1M steps of 512 sequences of 1,024 tokens, 524.3B tokens, plus distillation. ELF-B trains for 5 epochs of OpenWebText: 45.2B tokens by the paper's Table 8, or 49.8B by its step count, about a tenth either way. The authors report that training longer did not improve ELF further.

For conditional generation the source text is encoded by T5, prepended to the noisy target as clean embeddings that are never noised, and dropped (zeroed) for 10% of training examples so that standard CFG on the source is available. With a 64-step ODE sampler, self-conditioning scale 1 and source-guidance scale 2, ELF-B scores 26.4 BLEU on WMT14 German-to-English, against 25.2 for a 99M autoregressive model and 24.8 for E2D2 (both numbers from the E2D2 paper) and 24.9 for CDCD. On XSum summarization it scores ROUGE-1/2/L of 36.0/12.2/27.8 against 33.4/11.6/25.8 for MDLM and 30.5/10.2/24.4 for the autoregressive model, all baselines reproduced by the authors. The generated outputs are 64 tokens, and the autoregressive baselines are small (99M); neither result is a comparison with a production translation or summarization system.

Limitations

The unconditional comparisons are in Gen. PPL and unigram entropy only, and ELF's best numbers sit at lower entropy than the baselines'. The paper has no held-out likelihood, because computing a flow model's likelihood takes an integral of the velocity field's divergence along each trajectory, so it cannot be compared with autoregressive models on perplexity of real text. The largest model is 652M parameters, and the scaling result (Figure 6) covers three sizes.

Generation length is fixed by the array shape: 1,024 positions for OpenWebText, 64 for the conditional tasks. The method depends on a pretrained encoder for its embedding space, and the authors' ablations show noticeably worse results without a contextual one. Few-step generation still needs distillation: Appendix B reports that ELF+PD below 8 steps loses quality, with Gen. PPL rising from 23.18 at 8 steps to 34.33 at 4 and 136.10 at 1.