Whisper: Robust Speech Recognition via Large-Scale Weak Supervision

One plain Transformer, trained on 680,000 hours of noisy web transcripts, transcribes new audio without fine-tuning.

Whisper skips self-supervised pre-training. OpenAI filtered captions and transcripts from the web, trained an ordinary encoder-decoder to predict them, and scored the model on test sets whose training data it never used.

Two recognizers tie at 2.7% word error on LibriSpeech. One was fine-tuned on LibriSpeech; the other, Whisper, used no benchmark's training data, and across 13 other test sets it makes 55.2% fewer errors on average.

Before Whisper, the strongest speech recognizers were built in two stages. First, a network learned from a very large amount of audio with no transcripts. The best-known method, wav2vec 2.0, hides stretches of the signal and trains the network to identify what was hidden, which teaches it the structure of speech sounds. Second, because that network has no way to output text, it was fine-tuned on a labeled dataset such as LibriSpeech (about 1,000 hours of read audiobooks). With all of LibriSpeech's labels, wav2vec 2.0 reached 1.8% word error on LibriSpeech test-clean.

That recipe has two costs. Every new deployment needs its own fine-tuning, with its own labeled data. And a model fine-tuned on one dataset learns that dataset's microphones, speakers and transcript conventions, so a system that is better than a human on LibriSpeech can make far more errors than a human on a phone call or in a meeting room.

Whisper (Radford et al., OpenAI, December 2022) removes both stages. It collects 680,000 hours of audio that people on the internet had already transcribed, filters out the worst of it, trains one standard encoder-decoder Transformer to predict the transcripts, and evaluates on benchmarks without ever training on their data. The architecture and loss are standard; the paper's results come from the data, the scale, and a token format that puts transcription, translation, timestamps and language identification into one decoder. The sections below follow that order: where the labels come from, what the model receives as input, the token format, the loss, how errors are counted, the robustness result, what scale buys, and how long recordings are decoded.

Three sources of labels

A speech dataset has two properties that trade against each other: how many hours it has, and how accurate its transcripts are.

Gold supervision means transcripts checked by people. SpeechStew, a strong example from 2021, mixes seven existing datasets for a total of 5,140 hours. The labels are good, but human checking is slow and expensive, and 5,140 hours is small next to the amount of audio available.

Self-supervision needs no transcripts, so it scales to whatever audio exists; the wav2vec 2.0 line of work reached 1,000,000 hours of unlabeled audio (BigSSL). What it learns is a representation of the audio. Because the training never pairs sound with text, the network has no decoder that writes words, and the paper names this as the reason these systems need a fine-tuning stage before they are useful.

Weak supervision, the paper's term for its own data, sits between the two. Captions, subtitles and posted transcripts are real text aligned to real audio, so a model trained on them learns to write text directly. They are also partial, inconsistent and sometimes wrong, because nobody checked them for training. As an analogy, gold supervision is a textbook with an answer key, short and correct; self-supervision is listening to a lot of radio in a foreign language, which teaches the sounds but not the spelling; weak supervision is watching thousands of subtitled films, where the subtitles are sometimes loose or missing. The analogy breaks in one place. A person who learned from radio can still write, while a self-supervised network has no text output until fine-tuning attaches one.

Choosing the weak corner has two consequences for the rest of the paper. Real labels mean the model can transcribe as soon as training ends, with no fine-tuning stage. The noise means most of the engineering effort goes into filtering the data before training.

Building the 680,000 hours

The final dataset has 680,000 hours of audio paired with text, in three parts. 117,000 hours are transcription in 96 languages other than English. 125,000 hours are audio in some language paired with an English translation, which becomes speech-translation training data. The remaining 680,000 − 117,000 − 125,000 = 438,000 hours are English transcription (the paper prints the first two numbers; the 438,000 is the subtraction).

Many of the bad labels come from other speech recognizers: many captions on the internet are the output of an older automatic speech recognition (ASR) system, and training on them would teach Whisper to imitate that system's mistakes and style, which the paper calls "transcript-ese". Such output has detectable habits. ASR systems usually produce a restricted form of written language: no capitalization, or all of it; no commas, exclamation marks or question marks; no paragraph breaks. A person transcribing by hand almost never writes a whole transcript in one case. So the filters drop all-uppercase and all-lowercase transcripts, and transcripts that never use the punctuation a person would.

A second filter checks language. An audio language detector (a model fine-tuned on the VoxLingua107 dataset) labels the spoken language, a text detector (CLD2) labels the transcript's language, and pairs that disagree are dropped, with one exception: if the audio is not English and the transcript is English, the pair is kept as an X → en translation example. Those pairs are the 125,000 hours of translation data. Then fuzzy de-duplication removes repeated transcripts, and transcripts that overlap the evaluation sets (TED-LIUM 3 in particular) are removed so the zero-shot scores are not contaminated. The whole pass looks like this, with an example input on each line:

# the web-transcript filters, applied to candidate (audio, transcript) pairs
keep = []
for audio, text in candidates:
    if text.isupper() or text.islower():     # "SO WHERE WERE WE" -> machine output
        continue
    if never_has_commas_or_marks(text):      # rule-based ASR punctuation -> drop
        continue
    spoken  = voxlingua107_detector(audio)   # e.g. "es"
    written = cld2_language(text)            # e.g. "en"
    if spoken != written:
        if written == "en":                  # Spanish audio, English text
            keep.append((audio, text, "translate"))   # -> X->en example
        continue                             # any other mismatch: drop
    keep.append((audio, text, "transcribe"))
keep = fuzzy_dedup(keep)                     # repeats and boilerplate
keep = remove_overlap_with_eval_sets(keep)   # e.g. TED-LIUM 3 talks

The authors also ran a second pass after training a first model. They computed its error rate on each training data source and manually inspected the sources with a high error rate and a large size. That inspection found partially transcribed audio, misaligned transcripts, and machine-generated captions the heuristics had missed, and those sources were removed.

Audio files are then cut into 30-second segments, each paired with the part of the transcript that falls inside it. Segments with no speech are kept too, sampled with lower probability than speech, and labeled as no-speech, which trains the model to detect voice activity. The transcripts are otherwise left as written: there is no step that standardizes numbers, casing or punctuation. The paper relies on the sequence-to-sequence model to learn the mapping from sound to written form, which also means Whisper's output needs no separate inverse text normalization step (the step that turns "ten dollars" into "$10" in a conventional pipeline).

From audio to 1,500 vectors

The authors wanted to measure what data and scale buy, so they used an off-the-shelf architecture: the encoder-decoder Transformer of Attention Is All You Need, with no conformer (the convolution-augmented Transformer common in speech at the time) and no new attention variant. A novel architecture would make it impossible to tell whether an improvement came from the data or from the model. The encoder turns 30 seconds of audio into a sequence of vectors; the decoder writes tokens while attending to them.

The encoder does not take the waveform. All audio is resampled to 16 kHz, so a 30-second segment is 480,000 samples. A short-time Fourier transform runs over 25 ms windows (400 samples) every 10 ms (160 samples), which gives 3,000 frames. Each frame's spectrum is pooled into 80 Mel bands, spaced the way human pitch perception is (narrow bands at low frequencies, wide bands at high ones), and the log of the magnitude is taken, because perceived loudness is roughly logarithmic in energy. This produces an 80 × 3,000 array of energy over frequency and time, globally scaled to lie between −1 and 1 with roughly zero mean over the training set. Shorter audio is padded with silence to 30 seconds in the released code.

A stem of two convolution layers, each with a filter width of 3 and a GELU activation, comes next. The first maps the 80 channels to the model width $d$. The second has a stride of 2, so it halves the time axis: 3,000 frames become 1,500 positions, one every 20 ms. Sinusoidal position embeddings are added, and the Transformer blocks (pre-activation residual blocks, with a final layer normalization on the encoder output) process the sequence. For Whisper Large, $d = 1280$, so the decoder attends over a 1,500 × 1,280 matrix for every 30-second segment. Hover over the spectrogram below to see how a moment in the window maps to a mel frame and then to an encoder position.

The 20 ms spacing of encoder positions reappears in the next section: Whisper's timestamp tokens are quantized to the nearest 20 ms, which the paper describes as the native time resolution of the model.

The decoder uses learned position embeddings and ties its input and output token embeddings (the same matrix turns a token into a vector and a vector into token scores). Text is tokenized with GPT-2's byte-level byte-pair encoding (BPE); for the multilingual models the vocabulary is refit to the multilingual data at the same size, so that other languages are not split into many short pieces. The encoder and decoder have the same width and the same number of blocks. The paper trains five sizes (Table 1), where the layer count is per side:

• Tiny: 4 layers, width 384, 6 heads, 39M parameters.
• Base: 6 layers, width 512, 8 heads, 74M.
• Small: 12 layers, width 768, 12 heads, 244M.
• Medium: 24 layers, width 1024, 16 heads, 769M.
• Large: 32 layers, width 1280, 20 heads, 1,550M.

Large therefore has 32 encoder blocks and 32 decoder blocks. Every model uses 64-dimensional attention heads (384/6, 512/8, and so on up to 1,280/20).

Every task as tokens

A usable speech system does more than recognize words. It detects whether anyone is speaking (voice activity detection), identifies the language, aligns words to times, and sometimes translates; in a conventional system each of these is a separate component. Whisper's decoder is an audio-conditional language model: it predicts text one token at a time, conditioned on the encoded audio and on the tokens before. The paper uses that to fold every task into the token sequence the decoder already predicts, with special tokens that say which task and format are wanted. One set of weights then serves all tasks and all languages, and a new behavior is a new token sequence, not a new model.

The sequence starts with <|startoftranscript|>. The model then predicts the spoken language as one of 99 language tokens (the language labels in training come from the VoxLingua107 detector), or <|nospeech|> if the segment has no speech. A task token follows, <|transcribe|> or <|translate|>, and then <|notimestamps|> if timestamps are not wanted. The text comes next. In timestamp mode, a start-time token precedes each caption segment and an end-time token follows it. Times are relative to the 30-second window and quantized to 20 ms, so the released tokenizer has 1,501 timestamp tokens, from <|0.00|> to <|30.00|>. The sequence ends with <|endoftranscript|>. Because a timestamp is a token like any other, predicting when a phrase starts uses the same mechanism as predicting the phrase, and no separate alignment model is needed.

During training, with some probability, the transcript of the preceding audio is also placed before <|startoftranscript|>, after a <|startofprev|> token, so the model can use longer text context to resolve an ambiguous word. The model reads that context but is not trained to predict it. Toggle the controls below to build each of these sequences.

Written out, the target for 2.4 seconds of Spanish with timestamps is this list:

# the decoder's target for one 30 s segment of Spanish, with timestamps
<|startoftranscript|>      # begin
<|es|>                     # spoken language, one of 99 language tokens
<|transcribe|>             # the task; <|translate|> means "write English"
<|0.00|> el zorro veloz <|2.40|>    # text between a start and an end time
<|endoftranscript|>
# no timestamps wanted: <|notimestamps|> after the task token
# silence: <|startoftranscript|> <|nospeech|> <|endoftranscript|>

In timestamp mode, if the last caption in a window is cut off by the 30-second boundary, the model predicts only its start-time token. That tells the long-form decoder (covered in the last sections) where to begin the next window. Translation uses the same machinery: <|translate|> makes the decoder write English text for non-English audio, trained on the X → en pairs the language filter kept.

The training objective

Write $\mathbf{x}$ for the log-Mel spectrogram of a segment and $u_1, \dots, u_T$ for its target tokens. The decoder is trained to predict each token from the audio and the tokens before it, with the standard cross-entropy loss:

The sum skips the positions of the previous-text context ("prev"), which is the loss mask from the last section. Every other token is a prediction target: the language token, the task and format tokens, the timestamps and the words. For the Spanish example above, the model is graded on predicting <|es|> after the start token, <|0.00|> before the first word, "zorro" after "el", <|2.40|> after "veloz", and so on. There is no contrastive term, no self-supervised auxiliary loss, and no separate alignment model. Here is one step with shapes, for Large:

# one training step, Whisper Large (width d = 1280), batch B = 256
mel   = log_mel(audio)                 # (B, 80, 3000): 80 channels x 3,000 frames
h     = gelu(conv1(mel))               # (B, 1280, 3000), kernel 3
h     = gelu(conv2(h))                 # (B, 1280, 1500), kernel 3, stride 2
h     = h.T + sinusoid_positions       # (B, 1500, 1280)
audio = encoder_blocks(h)              # (B, 1500, 1280), 32 blocks

tokens = [prev..., sot, lang, task, ts, words..., ts, eot]  # one row per segment
logits = decoder(tokens[:-1], audio)   # (B, T-1, vocab), 32 blocks + cross-attention
loss   = cross_entropy(logits, tokens[1:], mask=not_prev_text)
loss.backward()                        # every parameter gets a gradient
adamw_step(clip_grad_norm=True)

Nothing in (1) accounts for noisy labels; it is the loss one would use on gold transcripts. The paper handles noise before training, with the filters, and relies on scale during training. One way to read why that works, which is this page's interpretation and not a claim the paper tests: a wrong transcript pushes the parameters in a direction specific to that transcript, while the correct mapping from sounds to words pushes the same way in millions of examples, so the consistent part dominates the average gradient.

The run itself is short. Each model trains for $2^{20}$ = 1,048,576 updates with a batch of 256 segments, which the paper puts at between two and three passes over the dataset. The optimizer is AdamW (Adam with decoupled weight decay) with gradient-norm clipping, a learning rate warmed up over the first 2,048 updates and then decayed linearly to zero, FP16 arithmetic with dynamic loss scaling, and activation checkpointing to save memory. The original models use no data augmentation and no regularization; with so few passes over so varied a dataset, the authors rely on the data itself to prevent overfitting.

Two later changes affect the reported results. Whisper Large-V2, released after the first models, was trained for 2.5 times more epochs with three regularizers added: SpecAugment (masking random frequency bands and time spans of the spectrogram), stochastic depth (randomly skipping residual blocks during training) and BPE dropout (randomly splitting words into different subword pieces). Several of the paper's headline tables, including the comparison with wav2vec 2.0 below, use Large-V2. Separately, early models tended to transcribe plausible but almost always wrong guesses of speaker names, a habit learned from web transcripts that label who is talking; a brief extra fine-tune on transcripts without speaker annotations removed it.

Counting errors

Every number in the rest of the paper is a word error rate (WER). Align the model's output (the hypothesis) with the reference transcript word by word, using the fewest edits. Count the substitutions $S$ (a reference word replaced by a different word), deletions $D$ (a reference word missing from the output) and insertions $I$ (an output word with no reference word), and divide by the number of reference words $N$:

The denominator counts only reference words, while insertions are unbounded, so WER can exceed 100%. The metric also treats every mismatch the same way. If the reference says "mister" and the model writes "Mr.", that counts as a substitution, even though a human reader would call the transcript correct. A zero-shot model has never seen a benchmark's transcript conventions, so it is hit hardest by this. Pick an example below and toggle the normalizer.

In the first example every word was heard correctly, and the raw score is still 9 errors over 11 reference words, 81.8%. After normalization both strings read "mister smith says you are late it cost $10000" and the WER is 0. In the second, normalization removes the style differences ("we'll" versus "We will", the capital letter, the period) and leaves the real mistake, "station" heard as "nation": 1 error over 8 words, 12.5%. The count $N$ changed from 7 to 8 because the normalizer expanded the reference's contraction. The third example is a repeat loop: two reference words and six extra output words, 300% even after normalization.

The English normalizer (Appendix C of the paper) removes text in brackets and parentheses, drops filler words such as "hmm" and "uh", expands contractions, removes most punctuation and diacritics, rewrites spelled-out numbers and currencies as digits ("ten thousand dollars" becomes "$10000"), and converts British spellings to American. For other languages it only removes bracketed text and punctuation and lowercases; for Chinese, Japanese, Thai, Lao and Burmese, which do not separate words with spaces, it puts a space between every character, so the score becomes a character error rate.

On several datasets normalization lowers WER by up to 50%, usually because of one quirk, such as a reference that splits contractions from words with a space. The normalizer was built by inspecting Whisper's own outputs, so it could flatter Whisper specifically, and the paper says so. To check, the authors re-scored with an independent normalizer from the FairSpeech project. On most datasets the two gave similar reductions for Whisper and for other open-source models; on WSJ, CallHome and Switchboard, Whisper's normalizer helped Whisper significantly more, because it forgives contractions ("you're" versus "you are") and written versus spoken numbers ("sixty-eight million dollars" versus "$68 million"). The code is released so others can score their models the same way. The comparisons below survive this check, but their absolute values depend on the normalizer.

Effective robustness

In 2015, Deep Speech 2 reported human-level accuracy on LibriSpeech test-clean: 5.3% WER for the system against 5.8% for a human transcriber. By 2022 the best published result was 1.4%, a further 73% reduction. Yet recognizers trained on LibriSpeech still made many more errors than people on other kinds of audio.

The paper's explanation is that the two scores measure different skills. The human who transcribed LibriSpeech had no special training on LibriSpeech, so their score measures generalization to audio they were not prepared for (out-of-distribution). A model trained on LibriSpeech's training split is tested on more of the same distribution, so its score measures in-distribution generalization, and part of what it learned is LibriSpeech itself: its readers, microphones and transcript style. The benchmark cannot tell those two kinds of accuracy apart. A model that never trained on LibriSpeech is graded the way the human was.

To measure the difference, the paper borrows effective robustness from Taori et al. (2020), the same measure the CLIP paper used for image classifiers. Across many models trained on a reference dataset, out-of-distribution error is well predicted by error on the reference dataset, and the models lie near one trend line. A model has positive effective robustness if it lands below that line: it does better on other datasets than its reference score predicts. The paper's Figure 2 shows that supervised LibriSpeech models make roughly twice as many errors as a human on other datasets, while the zero-shot Whisper models lie close to the human, with the human's 95% confidence interval inside Whisper's robustness frontier.

The cleanest version of the comparison holds the reference score fixed. Table 2 of the paper takes the supervised wav2vec 2.0 Large model (no language model) and Whisper Large-V2, which both score 2.7 WER on LibriSpeech Clean, and compares them on 13 other datasets.

Over the 13 other datasets, wav2vec 2.0 averages 29.3 WER and Whisper averages 12.8. The paper reports the result as a 55.2% average relative error reduction, where each dataset's reduction is (wav2vec − Whisper) / wav2vec and the 55.2 is the mean of the 13 reductions. (Computing the reduction of the two averages instead gives (29.3 − 12.8) / 29.3 = 56.3%.) The per-dataset reductions range from 74.7% on Artie (24.5 → 6.2) and 69.9% on Common Voice (29.9 → 9.0) down to 46.2% on AMI SDM1, a far-field meeting microphone, and only 16.1% on LibriSpeech Other (6.2 → 5.2). LibriSpeech Other is the harder half of the same audiobook corpus that wav2vec 2.0 was fine-tuned on, so it is the dataset where the fine-tuned model is least out of its distribution.

The effect also shows at small scale. The smallest Whisper, Tiny with 39M parameters, scores 6.7 WER on LibriSpeech test-clean, a weak number for 2022, and is still roughly competitive with the best supervised LibriSpeech model when both are evaluated on other datasets.

Additive noise is a second test. The paper mixes white noise and pub noise (recorded crowd and bar sounds, from the Audio Degradation Toolbox) into LibriSpeech audio at controlled signal-to-noise ratios (SNR, in decibels; lower is noisier). At 40 dB SNR many LibriSpeech models beat zero-shot Whisper. Their error rises quickly as the noise increases, and under pub noise they fall behind Whisper below about 10 dB.

The paper also compares Whisper with professional transcription directly. On 25 recordings from the Kincaid46 dataset, transcribed by Whisper and by five services (one computer-assisted, four purely human), the computer-assisted service had a WER 1.15 points lower than Whisper's, and the purely human services were lower by only a fraction of a point. The paper's wording follows those numbers: Whisper "approaches" human accuracy and robustness.

Model size and data size

If the result comes from data, two questions follow: does performance keep improving with a bigger model, and with more data?

Across the five model sizes, zero-shot performance keeps improving for multilingual recognition, translation and language identification. English recognition shows diminishing returns, which the paper suggests may come from approaching human-level accuracy.

For data size, the authors trained the Medium model on 0.5%, 1%, 2%, 4% and 8% of the dataset and on all of it (Table 6), selecting each checkpoint by validation loss. The figure below plots the table. Look at the last interval, from 54,486 to 681,070 hours: 12.5 times more data.

English WER falls quickly at first: 30.5 at 3,405 hours, 14.4 at 13,621 hours. It then slows, to 10.9 at 54,486 hours, and the full 681,070 hours buy one more point, to 9.9. Multilingual WER follows a power-law trend (a straight line on log-log axes) up to 54,486 hours and then improves by only 7 more points, from 36.4 to 29.2. Translation is near zero up to 6,811 hours (0.2 and 1.7 BLEU), improves roughly linearly in log-hours to 19.2 BLEU at 54,486 hours, and reaches 24.8 on the full set. The paper offers two readings of the flattening at the end, without choosing between them: the largest models may be under-trained relative to the dataset, and longer training plus bigger models would help; or speech recognition may be nearing the end of the gains from more data.

The multilingual results also allow a comparison across languages. For each language, the paper plots Whisper's WER on the Fleurs benchmark against the hours of that language in the training set, on log-log axes. The squared correlation coefficient between log WER and log hours is 0.83, and WER halves for every 16-fold increase in data:

Equation (3) is the paper's "halves every 16×" rewritten as a slope of −1/4 on log-log axes. A language at 40% WER with 100 hours would be predicted at 20% with 1,600 hours and 10% with 25,600 hours. Drag the slider to walk along the line.

The languages furthest above the line, meaning worse than their data predicts, are Hebrew (HE), Telugu (TE), Chinese (ZH) and Korean (KO). The paper notes that they use unique scripts and are more distantly related to the Indo-European languages that make up most of the training data, and it also names a poor BPE fit and data quality as possible causes. Because WER tracks data this closely, the direct remedy for a weak language is more data, and most languages have less than 1,000 hours because the collection pipeline was English-centric.

Training one model on 99 languages and several tasks raises the question of interference, called negative transfer: the tasks compete for capacity and each ends up worse than a dedicated model. The paper finds it only at small scale. Small joint models underperform English-only models trained with the same compute. The largest joint models outperform their English-only counterparts on English, a positive transfer, although about 65% of the joint model's compute goes to English recognition.

On speech translation, Whisper sets a new zero-shot state of the art of 29.1 BLEU on CoVoST2 X→en (BLEU scores n-gram overlap with reference translations; higher is better). The paper credits its 68,000 hours of X→en data for those languages, against the 861 hours of CoVoST2's own training set. On multilingual recognition, Whisper beats XLS-R, mSLAM and Maestro on Multilingual LibriSpeech in the zero-shot setting but significantly underperforms prior work on VoxPopuli. On language identification, it trails the supervised state of the art on Fleurs by 13.6%. Part of that gap is structural: 20 of Fleurs' 102 languages have no Whisper training data, which caps accuracy at 80.4%, and on the 82 overlapping languages Whisper reaches 80.3%.

Long audio and decoding

As models get larger, perception errors such as confusing similar-sounding words steadily decrease. The errors that remain, especially in long recordings, are of a different kind, and the paper describes them as a combination of failure modes of sequence-to-sequence models, language models and text-audio alignment: getting stuck in a repeat loop, not transcribing the first or last few words of a segment, and complete hallucination, where the output is unrelated to the audio. The decoder is trained to produce fluent text, so when the audio gives it little to go on it can continue writing fluent text anyway.

Long audio makes these errors worse. Whisper sees 30 seconds at a time, so a long recording is transcribed by decoding one window, shifting the window forward by the last timestamp the model predicted, and repeating, with the previous window's text given as context. A wrong timestamp shifts the next window to the wrong place, and a bad transcript becomes the next window's context. The paper adds no new model for this; Section 4.5 lists decoding heuristics, each of which lowered WER overall in Table 7, though not evenly across datasets:

# long-form transcription, 30 s window at a time (paper Sec 4.5, Table 7)
t0 = 0.0
while t0 < audio_length:
    window = mel(audio[t0 : t0 + 30])
    for temp in [0.0, 0.2, 0.4, 0.6, 0.8, 1.0]:
        prompt = prev_text if temp < 0.5 else None
        seg = decode(window, temp, prompt,        # beam search, 5 beams, at 0.0
                     first_timestamp_max=1.0)     # start within 0-1 s
        looping  = gzip_ratio(seg.text) > 2.4
        unsure   = seg.avg_logprob < -1.0
        if not (looping or unsure):
            break                                 # accept this temperature
    if seg.nospeech_prob > 0.6 and seg.avg_logprob < -1.0:
        seg = SILENCE                             # both signals must agree
    emit(seg)
    prev_text = seg.text
    t0 += seg.last_timestamp                      # shift by predicted time

The first check is the compression ratio: the size of the decoded text in bytes divided by its size after compression (the released code uses zlib, the same DEFLATE algorithm as gzip). Repetitive text compresses well. With the released function, a normal 167-byte transcript ("So where were we? Right, the budget. We moved the launch to March...") has a ratio of 1.35, while "Thank you." written 16 times has a ratio of 7.95. A ratio above 2.4 marks the segment as a likely repeat loop. The second check is the average log-probability of the generated tokens; below −1, the model is treated as guessing.

When either check fails, the segment is decoded again at a higher temperature. Temperature divides the decoder's token scores before the softmax: at 0 the most probable token is always chosen, and higher values spread probability across more tokens. Decoding starts at 0 with beam search over 5 beams (keeping the 5 most probable partial transcripts at each step instead of only one), which by itself reduces looping compared with greedy decoding. If a check fails, the temperature rises by 0.2, up to 1.0; in the released code, temperatures above 0 use sampling with 5 candidates instead of beam search. The randomness moves the decoder off the high-probability cycle it was stuck in.

Three smaller rules complete the procedure. The previous window's text is given as context only when the temperature is below 0.5, so a window that needed a high temperature does not feed its uncertain text forward. A window is declared silent only when the <|nospeech|> probability is above 0.6 and the average log-probability is below −1; the paper found the no-speech probability alone unreliable. The first timestamp is constrained to lie between 0 and 1 second, which prevents the model from skipping the first words of a window. Move the temperature slider in the figure below to see the compression-ratio guard trigger at low temperature and stop triggering at higher ones.

The high-temperature setting shows the limit of the compression check. Random, fluent, wrong text does not repeat, so its ratio stays low; the average log-probability check is the one meant to catch it, and the paper calls these heuristics a workaround for noisy predictions that more research should replace.

What is still open

The paper's limitations section lists the next steps. Low-resource languages need more data, since most have less than 1,000 hours and the scaling trend in Figure 8 predicts their error directly from their hours. Fine-tuning remains unstudied: the paper evaluates only zero-shot, and fine-tuning on high-quality supervised data where it exists would likely lower error further. Reinforcement learning on the decoder could optimize decoding directly and reduce the loops and hallucinations of the previous section. The paper suspects that part of Whisper's robustness comes from its decoder, an audio-conditional language model, but cannot say how much comes from the encoder, the decoder or both. It proposes two ways to find out: train a decoder-less model with a CTC loss (a per-frame output layer that needs no language model), or pair existing encoders such as wav2vec 2.0 with a language model and measure the change. Last, Whisper uses no unsupervised pre-training or self-training; the authors did not find them necessary, and adding them as auxiliary objectives might still help.

The recipe, with its numbers: 680,000 hours of filtered web transcripts, cut into 30-second segments; an 80 × 3,000 log-Mel input reduced to 1,500 encoder positions; a standard Transformer with up to 32 blocks per side and 1,550M parameters; one decoder that predicts language, task, timestamps and text as tokens, trained with plain cross-entropy for 2²⁰ updates; scored zero-shot with a released normalizer. On LibriSpeech Clean it ties a fine-tuned wav2vec 2.0 at 2.7 WER; on 13 other datasets it averages 12.8 against 29.3.