wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations

Pretrain a speech model on 53,000 hours of untranscribed speech by hiding short spans of the audio and training it to pick each hidden frame's true discrete code out of 100 distractors. Ten minutes of transcribed audio then fine-tunes it to 4.8% word error rate on LibriSpeech test-clean.

A convolutional encoder turns raw audio into 49 frames per second, a Transformer reads the frames with about half of them masked, and a jointly learned codebook supplies the targets. This page walks through the encoder, the masking, the quantizer, and the loss with the paper's constants, then one training step in full.

In 2020 a strong English speech recognizer trained on 960 transcribed hours, and almost none of the world's roughly 7,000 languages has that much transcribed speech.

The 960 hours are LibriSpeech, read audiobooks with transcripts, and the best systems of the time added tens of thousands of hours more through pseudo-labeling. Untranscribed audio, on the other hand, is cheap everywhere. wav2vec 2.0 asks how much of a recognizer can be learned from audio alone, with transcripts used only at the end.

The recipe is the one BERT used for text, adapted to a signal with no vocabulary. Encode the waveform into a sequence of latent frames. Hide about half of them. Ask a Transformer to work out, from the unmasked neighbors, what was hidden. Since a hidden frame is a continuous vector rather than a word, the model also learns a discrete codebook, and the task becomes: among 101 candidate codewords, which one belongs at this masked position? After pre-training on 53,000 hours of LibriVox audio, a linear layer and a Connectionist Temporal Classification (CTC) loss on top of the Transformer turn it into a recognizer. With all 960 labeled hours it reaches 1.8% word error rate (WER) on test-clean and 3.3% on test-other. With ten minutes of labeled audio, 48 recordings, it reaches 4.8% and 8.2%.

The page follows the data: the convolutional encoder, span masking, the Gumbel-softmax quantizer, the contrastive and diversity losses, then one complete training step with tensor shapes, then fine-tuning and the numbers. Every constant comes from the paper or from the authors' fairseq code, which is cited where the paper leaves a mechanism ambiguous.

Waveform to 49 frames per second

Audio at 16 kHz is 16,000 floats per second, and phonemes last tens of milliseconds, so a Transformer over raw samples would attend across hundreds of positions per sound. The feature encoder $f$ shrinks this first. It is a stack of seven one-dimensional convolutions, each with 512 output channels, followed by layer normalization and a GELU activation. Kernel widths are (10, 3, 3, 3, 3, 2, 2) and strides are (5, 2, 2, 2, 2, 2, 2). The product of the strides is 320 samples, so one output frame covers 20 ms of audio, and the receptive field works out to 400 samples, 25 ms. One second of audio becomes 49 frames (the last partial window is dropped), which the paper reports as a 49 Hz frame rate.

The waveform is normalized to zero mean and unit variance before the first convolution (the LibriSpeech-only BASE model instead normalizes the output of the first convolution and drops the per-block layer normalization). The paper calls the encoder's final output the latent speech representations, a matrix $\mathbf{z}_1,\dots,\mathbf{z}_T$ with $T$ rows of 512 numbers. For a 15.6 second training crop of 250,000 samples, $T = 781$. Those latent frames go two places. A linear layer lifts them from 512 to the Transformer width (768 for the BASE model, 1,024 for LARGE) and they enter the context network. Unchanged copies of the same frames also go to the quantizer, which produces the training targets.

Hover a frame in the figure to see which 25 ms of the waveform it summarizes, and step the depth slider to watch the stride compound through the seven blocks:

The context network $g$ is a standard Transformer encoder: 12 blocks with 8 heads and a 3,072-wide feed-forward layer for BASE (about 95M parameters in total), 24 blocks with 16 heads and 4,096-wide feed-forward for LARGE (about 317M). Instead of fixed sinusoidal position embeddings, a grouped convolution with kernel 128 and 16 groups runs over the input sequence, is passed through a GELU, and is added to the input. A convolution can only see relative offsets, so this acts as a relative position code, and the Transformer's output $\mathbf{c}_1,\dots,\mathbf{c}_T$ has one 768-dimensional vector per frame, each computed with attention over the whole crop.

Masking spans of latent frames

Masking happens after the encoder and before the Transformer, on the latent frames, never on the waveform. The procedure has two constants. Every frame is a candidate start with probability $p = 0.065$, starts are sampled without replacement, and each start masks itself and the next $M - 1$ frames with $M = 10$. Spans may overlap. A masked frame's 768-dimensional input vector is replaced by a single learned vector shared by every masked position (fairseq calls it mask_emb), the analogue of BERT's [MASK] token.

The two constants imply the coverage. A frame stays unmasked only if none of the 10 candidate start positions ending at the frame itself fired, so the masked fraction is about $1 - (1 - 0.065)^{10} \approx 0.49$. Because spans overlap and merge, the average contiguous masked run is longer than 10: the paper reports a mean of 14.7 frames, or 299 ms, with a median of 10 and a maximum near 100 frames in a 15 second crop. A simulation with the fairseq sampler (starts sampled without replacement, $\lfloor pT + u \rfloor$ of them) reproduces 49.1% and 14.7.

Why spans instead of individual frames? Adjacent 20 ms frames are nearly identical, so a single masked frame can be filled in by copying a neighbor, and the pretext task (the masked-prediction problem the model trains on before it sees any transcript) teaches nothing. The ablation in the paper's Appendix A confirms it: with span length 1 the self-supervised accuracy goes up and the downstream recognizer gets worse. Drag both sliders to the paper's values and read the coverage, then push $M$ to 1 and to 20:

The quantizer: two codebooks of 320

A masked-prediction loss needs a target. For text the target is a word index and the loss is a softmax over the vocabulary. A 512-dimensional speech frame has no index, so wav2vec 2.0 builds one with a learned product quantizer: $G = 2$ codebooks (the paper says groups), each holding $V = 320$ entries, each entry a vector of 128 numbers for BASE. A frame is quantized by picking one entry from each codebook and concatenating the two picks into a 256-dimensional vector, which a linear layer maps to the target $\mathbf{q}_t$. Two codebooks of 320 give $320^2 = 102{,}400$ distinct codewords from only $640$ stored vectors of 128 numbers; a single codebook with that many codewords would need 102,400 stored vectors of 256 numbers.

Picking an entry is an argmax, which has no gradient, and the codebook has to be learned jointly with everything else. The paper uses the Gumbel-softmax with the straight-through estimator. A linear layer maps the 512-dimensional frame $\mathbf{z}_t$ to $G \cdot V = 640$ logits $\mathbf{l} \in \mathbb{R}^{2 \times 320}$. For codebook $g$, the probability of entry $v$ is

where $\tau$ is a temperature and $n$ is Gumbel noise, one fresh draw per logit. The paper typesets the temperature as $\exp(l + n)/\tau$, outside the exponential, where it cancels between numerator and denominator and would have no effect. Equation (1) above puts it inside, which is what the Gumbel-softmax paper defines and what the code does (PyTorch's gumbel_softmax computes softmax((logits + gumbels) / tau)).

Adding Gumbel noise and taking the argmax is exactly a sample from the categorical distribution $\mathrm{softmax}(\mathbf{l}_g)$ (the Gumbel-max trick), so the pick is random when two logits are close and deterministic when one dominates. The forward pass uses the hard argmax $i = \arg\max_v p_{g,v}$ and returns the codebook entry $e_{g,i}$ as a one-hot selection. The backward pass differentiates the soft probabilities $p_g$ in place of the one-hot: PyTorch implements this as y_hard - y_soft.detach() + y_soft, which has the value of the hard pick and the gradient of the soft one. So the logits, the linear layer producing them, and through it the convolutional encoder all receive gradient from the loss on $\mathbf{q}_t$. The output is computed as $\sum_v y_{g,v}\, e_{g,v}$, where $y_g$ is the straight-through selection above (one-hot in value), so the sum equals the selected entry $e_{g,i}$ and that codebook vector receives gradient.

The temperature is annealed. It starts at 2, is multiplied by 0.999995 after every update, and is floored at 0.5 for BASE and 0.1 for LARGE. Reaching 0.5 from 2 takes about 277,000 updates, so BASE (400,000 updates) spends its last third at the floor. Early on the soft probabilities are flat, so many codebook entries get gradient for each frame; late, they approach the one-hot pick and the estimator's bias shrinks. Drag the temperature and resample the noise to see how often the argmax changes:

After pre-training the quantizer is discarded. Appendix D of the paper checks what it learned by running the LARGE model over TIMIT, a corpus with hand-labeled phonemes, and counting which phoneme co-occurs with each codeword. Many codewords fire for one specific phoneme, although no phoneme labels were used in pre-training; the silence label, which is 22% of TIMIT, is spread over many codewords.

The contrastive loss

For each masked position $t$ the Transformer emits a context vector $\mathbf{c}_t$. The quantizer emits the target $\mathbf{q}_t$ from the unmasked frame at the same position. The loss scores $\mathbf{q}_t$ against $K + 1$ candidates $\mathbf{Q}_t$: the true one plus $K = 100$ distractors, which are the quantized targets of other masked positions in the same utterance, sampled uniformly. With cosine similarity $sim(\mathbf{a},\mathbf{b}) = \mathbf{a}^{\top}\mathbf{b} / \lVert\mathbf{a}\rVert\lVert\mathbf{b}\rVert$ and a temperature $\kappa = 0.1$,

Read (3) as a 101-way classification problem whose logits are cosine similarities divided by 0.1, with the true codeword in class 0, and the loss is the cross-entropy of that classification. In the code, $\mathbf{c}_t$ is first projected from 768 to 256 dimensions by a linear layer so that it can be compared with the 256-dimensional targets; the paper does not mention this projection. If a distractor happens to be the identical codeword to the positive (two frames quantized the same way), the code sets its logit to $-\infty$ so it is not counted as a wrong answer. That rule has a consequence: the fewer distinct codewords the quantizer uses, the fewer effective distractors each masked frame faces, and a codebook collapsed to one entry would leave none and a loss of zero. The diversity loss in the next section rules this out.

This form is the InfoNCE loss from contrastive predictive coding (CPC), and it comes with a bound: for $N$ candidates including the positive, $I(\mathbf{c}_t;\mathbf{q}_t) \ge \log N - \mathcal{L}_m$. With $N = 101$ the constant is $\log 101 \approx 4.62$ nats, so a model whose loss averages 1.5 nats has, under the assumptions of the CPC derivation, at least 3.1 nats of mutual information between its context vector and the quantized frame it could not see. A model that gives every candidate the same score sits at $\mathcal{L}_m = \log 101$ and the bound says nothing. Distractors are drawn from the same utterance because distractors from other speakers would be easy to reject by voice alone, and the model would learn speaker identity instead of content. The ablation confirms that taking 50 of the 100 negatives from other utterances in the batch raises dev WER from 7.97 to 8.79.

The figure lays out one masked position: the positive and 100 distractors as points on the cosine axis, the softmax after dividing by $\kappa$, and the resulting loss next to the $\log(K+1)$ ceiling. Drag $\kappa$ toward 1 and watch the probabilities flatten; drag $K$ to 1 and the task becomes a coin flip:

The diversity loss

A quantizer trained only through (3) is rewarded for collapsing: every distractor that shares the positive's codeword is discarded, so a codebook using 20 of its 320 entries faces fewer effective distractors than one using all 320, and the task gets easier as the codebook shrinks. Appendix F reports what happens with the diversity term removed: dev WER rises from 7.97 to 8.48. The diversity loss, borrowed from Dieleman et al. (2018), maximizes the entropy of the average selection distribution. Let $\bar{p}_g$ be the softmax over codebook $g$'s logits (no Gumbel noise, no temperature) averaged over all masked frames in the batch. Then

Minimizing negative entropy maximizes entropy, and the entropy of a distribution over 320 entries is maximal when all 320 are used equally often. Because of the averaging, this penalizes uneven usage across the batch rather than confident picks within a frame: each frame can be confidently assigned to one entry while the batch as a whole spreads across all of them. The total pre-training objective is

The code implements a different function with the same minimizer. It computes each codebook's perplexity $\exp H(\bar{p}_g)$, the effective number of entries in use, and penalizes $(GV - \sum_g \exp H(\bar{p}_g)) / GV$. That is 0 when both codebooks use all 320 entries uniformly and about 0.997 when each has collapsed to one entry. The paper's footnote calls the two "equivalent"; they share a minimum but their gradients differ, and the perplexity version is what $\alpha = 0.1$ was tuned against. Setting $\alpha$ to 0.2 gives 8.58 with a large seed variance, which the authors describe as slight instability. Drag the usage histogram to concentrate mass on a few entries and read the entropy, perplexity, and both forms of the penalty:

One training step, with shapes

The pieces assembled for the BASE model on LibriSpeech, one crop of one utterance. Batches are built by cropping 250,000 samples (15.6 s) from each recording and packing crops up to 1.4 million samples per GPU; on 64 V100 GPUs that is about 1.6 hours of audio per update. Training runs 400,000 updates with Adam, a linear warm-up over the first 8% (32,000 updates) to a peak learning rate of $5 \times 10^{-4}$, then linear decay; it takes 1.6 days. LARGE uses 128 GPUs, 2.7 hours of audio per update, 250,000 updates for LibriSpeech (2.3 days) and 600,000 for LibriVox (5.2 days).

# one pre-training step, BASE model, one 15.6 s crop (250,000 samples)
wav = normalize(crop)                       # [250000]  zero mean, unit var
z   = conv_encoder(wav)                     # [781, 512] 20 ms per frame
q_in = z                                    # quantizer sees UNMASKED z
x   = post_proj(z)                          # [781, 768]
mask = sample_spans(T=781, p=0.065, M=10)   # ~382 masked frames (49%)
x[mask] = mask_emb                          # one learned 768-vector
c   = transformer(x)                        # [781, 768] full context
c_m = final_proj(c[mask])                   # [382, 256]

logits = weight_proj(q_in[mask])            # [382, 640] = 2 groups x 320
probs  = gumbel_softmax(logits.view(382, 2, 320), tau)   # hard one-hot fwd
e      = pick_codewords(probs)              # [382, 2, 128] -> cat -> [382, 256]
q_m    = project_q(e)                       # [382, 256] the targets

negs = sample_other_masked(q_m, K=100)      # [100, 382, 256] same utterance
sim  = cosine(c_m, cat([q_m, negs])) / 0.1  # [101, 382]  kappa = 0.1
L_m  = cross_entropy(sim.T, target=0)       # summed over 382 masked frames
L_d  = (640 - sum_g perplexity(mean_probs_g)) / 640   # codebook usage
loss = L_m + 0.1 * 382 * L_d + 10 * 382 * mean(z ** 2)  # fairseq weights
loss.backward()   # grads: transformer, projections, codebooks, encoder (x0.1)

Three things in the listing are code facts the paper does not spell out. The diversity loss (and, for LibriSpeech runs, the feature penalty described next) is multiplied by the number of masked frames: the contrastive loss is a sum over masked frames, not a mean, so this scaling keeps $\alpha = 0.1$ a per-frame weight. For the LibriSpeech-only runs an L2 penalty on the encoder output (weight 10 in fairseq) and a 0.1 multiplier on the encoder's gradient regularize the convolutional stack, whose 4 million parameters otherwise overfit to 960 hours; neither is used for the 53,000 hour LibriVox run. And the quantizer is evaluated on the unmasked features at the masked positions only, so $\bar{p}_g$ in (4) averages over the roughly 382 masked frames per crop, not all 781.

The gradient reaches every parameter: the Transformer and its final projection through $\mathbf{c}_t$; the codebook vectors, the logit layer, and (scaled by 0.1) the convolutional encoder through $\mathbf{q}_t$ and the straight-through estimator; and the encoder a second time through the Transformer's input. Appendix F reports that blocking the gradient from the quantizer into the encoder raises dev WER from 7.97 to 8.41, so the encoder needs both paths. Dropout of 0.1 is applied inside the Transformer, on the encoder output, and on the quantizer input, and whole Transformer layers are dropped with probability 0.05 (BASE) or 0.2 (LARGE) during LibriSpeech pre-training. The checkpoint with the lowest $\mathcal{L}_m$ on held-out audio is kept.

Fine-tuning with CTC

Pre-training produces a network that maps audio to 49 context vectors per second and has never seen a letter. To make a recognizer, a randomly initialized linear layer maps each $\mathbf{c}_t$ to $C$ output classes. For LibriSpeech these are characters: the paper counts 29 character tokens plus a word boundary token; the fairseq dictionary adds a blank and its own special symbols, for 32 outputs in total. The quantizer, its projection, and the contrastive head are removed.

Frames outnumber characters by roughly three to one (a 15.6 s crop has 781 frames and about 230 characters), and nobody knows which frames belong to which letter. Connectionist Temporal Classification (CTC) handles this without an alignment. It predicts one of the 32 symbols per frame, defines the output of a frame sequence by collapsing repeated symbols and then deleting blanks (so h h ε e l ε l l o reads hello, and the blank between the two l's is what lets a double letter survive), and the loss is the negative log of the total probability of every frame sequence that collapses to the transcript:

The sum has exponentially many terms and is computed exactly by a forward-backward dynamic program in the same spirit as HMM training. CTC treats the per-frame outputs as conditionally independent given the audio, which is why decoding pairs it with a language model.

# fine-tuning: characters from frames, CTC
z = conv_encoder(wav)        # frozen after pre-training
c = transformer(mask_aug(z)) # SpecAugment-style span masks on z
y = linear(c)                # [T, 32]: 26 letters, ', |, blank, ...
loss = ctc(log_softmax(y), transcript)   # sums over all alignments
# decode: greedy collapse, or beam search with a word-level LM

The fine-tuning schedule protects the pre-trained weights. The convolutional encoder is frozen throughout. For the first 10,000 updates only the new linear layer trains, so that random output weights do not send large gradients into the Transformer; then the Transformer trains too. The learning rate warms up for 10% of updates, holds for 40%, and decays for the remaining 50%. Peak rates are small: $2 \times 10^{-5}$ or $3 \times 10^{-5}$ for the Libri-light subsets (the 10 minute, 1 hour, and 10 hour label sets) and $10^{-4}$ for the 960 hour set. The number of updates scales with the data: 12,000 for ten minutes, 13,000 for one hour, 20,000 for ten hours, 50,000 for 100 hours, 320,000 for 960 hours. A SpecAugment-style augmentation masks spans of ten latent frames (reusing mask_emb) and spans of 64 channels (set to zero) during fine-tuning; the paper credits it with delaying overfitting on the smallest subsets, where 12,000 updates at 1,600 seconds of audio per batch amount to about 32,000 passes over the same 600 seconds of recordings.

Decoding adds a language model (LM) trained on the LibriSpeech text corpus: either a 4-gram model or a 20-block Transformer LM with width 1,280. Beam search combines the acoustic log-probabilities with a weighted LM score and a per-word insertion penalty; the two weights are tuned by Bayesian optimization on dev-other (128 trials), with beam 1,500 for the 4-gram and 500 for the Transformer at test time. The acoustic model emits characters while the LM scores words, which the paper notes delays LM feedback compared with the word-piece vocabularies used by competing systems.

Results: 10 minutes of labels

The headline experiments hold the pre-trained model fixed and vary the amount of transcribed audio: 10 minutes, 1 hour, 10 hours (the Libri-light subsets, drawn from LibriSpeech), 100 hours (train-clean-100), and all 960 hours. WER is reported on test-clean (read speech from easier speakers) and test-other (harder speakers and recording conditions). The pre-training corpus is LibriSpeech's own 960 hours without transcripts (LS-960) or 53,200 hours of LibriVox (LV-60k).

Test-clean / test-other WER in percent, all with the Transformer LM (Tables 1 and 2 of the paper); the last column is the best published row at each budget in Table 1. Three comparisons from that table. The paper takes iterative self-training (Noisy student) as the previous state of the art on 100 hours among systems that, like wav2vec 2.0 LS-960, use only LibriSpeech audio as unlabeled data: 4.2 / 8.6. The LARGE LS-960 model fine-tuned on the same 100 hours gets 2.3 / 5.0, a relative reduction of 45% and 42%. With one hour of labels the same model, at 3.9 / 7.6, is still ahead of that 100 hour system: 100 times less transcribed audio and the same unlabeled audio. And with ten minutes and LibriVox pre-training, 4.8 / 8.2 is better than every 100 hour system in Table 1 except iterative pseudo-labeling with LibriVox (3.72 / 7.11).

The language model does more work when labels are scarce. Without any LM, the ten minute LV-60k model is at 40.2 / 38.7; the 4-gram brings that to 6.6 / 10.3 and the Transformer LM to 4.8 / 8.2. At 960 hours the no-LM number is 2.2 / 4.5, and the Transformer LM only trims it to 1.8 / 3.3. The ten minute acoustic model has learned the sounds but has seen too few words to spell them; the error tables in Appendix E are dominated by phonetic spellings such as "are" transcribed as "ar", which a word-level LM corrects. Toggle the LM and the test set in the figure and read the three curves:

On the full 960 hours the LARGE LV-60k model's 1.8 / 3.3 is a new best on test-other and within 0.1 of Noisy student's 1.7 on test-clean. Trained from scratch without pre-training, the same LARGE network reaches 2.1 / 4.6, worse than the 1.9 / 4.1 of ContextNet, a purely supervised model trained on the same 960 labeled hours and the architecture Noisy student builds on. The LARGE network on its own is therefore not the source of the gain; pre-training is. On TIMIT phoneme recognition, fine-tuned without any LM, LARGE LS-960 gets a phone error rate of 7.4 on dev and 8.3 on test against 9.6 / 11.6 for the previous best, vq-wav2vec; the paper quotes this as a 23% / 29% relative reduction (the test figure computes to 28.4%).

Two more effects are visible in the table. Going from BASE to LARGE on the same LS-960 audio helps most on test-other, the noisier set (12.9 to 10.8 at ten minutes). Going from LS-960 to LV-60k, 55 times more unlabeled audio with the same LARGE architecture, helps at every label budget, and most at the smallest (10.8 to 8.2). Section 5.1 of the paper quotes the ten minute LV-60k result as 5.2 / 8.6; the abstract, the conclusion, and Table 1 all say 4.8 / 8.2, and the table is the number used here.

Why quantize targets, not inputs

The earlier pipeline, vq-wav2vec followed by a BERT over the discrete codes, quantized the frames once and fed the codes to the Transformer as input and used them as targets. wav2vec 2.0 keeps the Transformer input continuous and quantizes only the target. The paper tests all four combinations in a reduced setup: BASE pre-trained on LS-960 for 250,000 updates with $p = 0.075$, fine-tuned on 10 hours, three fine-tuning seeds, dev WER averaged over dev-clean and dev-other.

Continuous input with quantized targets, 7.97, wins. Quantizing the input as well, the vq-wav2vec design, gives 12.18: the Transformer loses whatever the codebook could not represent. Keeping everything continuous gives 8.58, and the paper reports that the training accuracy of picking the correct latent rises from 62% to 78% when the target is continuous. The pretext task got easier and the recognizer got worse, because a continuous target carries speaker and background detail that identifies the frame without any phonetic knowledge. Adding an MLP on the continuous target (8.51) or separate encoders for input and target (8.90) does not close the gap.

The remaining ablations in Appendices A and F are small. Mask length 8, 12, or 15 instead of 10 costs 0.2 to 0.5 WER. Sampling span lengths from a distribution instead of a fixed 10, without overlap, is worse in every variant tried (the best, Poisson with mean 15, gives 8.13). Doubling the negatives to 200 (8.12) or sampling them from any time step rather than masked ones (8.07) changes little. Removing the Gumbel noise gives 8.73 with a seed standard deviation of 0.42, twenty times the baseline's 0.02. Four codebooks of 18 entries (9.02) is worse and eight codebooks of 8 (8.13) is close. Widening the receptive field to 30 ms (7.99) or the positional convolution to kernel 256 (8.14) does nothing.

What the paper does not settle

The framework has costs and open ends the paper states or that follow-up work exposed. The acoustic model emits characters through CTC while the strongest contemporaries used word pieces and sequence-to-sequence decoders; the authors expect further gains from both and did not test them. The 1.8 / 3.3 result needs a Transformer language model and a beam of 500; the model on its own is at 2.2 / 4.5. Pre-training LARGE on LibriVox took 128 V100 GPUs for 5.2 days; most users start from the released checkpoints.

The contrastive objective itself was the first part to be replaced. HuBERT (2021) keeps the encoder, the span masking, and the Transformer, but predicts cluster assignments from an offline k-means over features with a plain cross-entropy, and matches or improves on wav2vec 2.0 at every label budget; data2vec (2022) predicts a teacher network's continuous layer averages and drops the codebook altogether. Both results suggest that what mattered in wav2vec 2.0 was masked prediction of a target derived from the unmasked signal, and that the quantizer plus InfoNCE was one workable way to make that target discrete rather than the essential one. Table 4 points the same way: fully continuous training still works, only 0.6 WER behind (8.58 against 7.97), far from the 12.18 of quantizing both.

The label-budget result stands regardless of which objective later won. A 5,760-fold drop in transcribed audio, from 960 hours to 10 minutes, costs 3.0 points of test-clean WER (1.8 to 4.8) after pre-training on 53,000 hours of untranscribed speech, and ten minutes of transcription is within reach for most of the 7,000 languages the paper's introduction counts.