AudioLM: a Language Modeling Approach to Audio Generation

Turn audio into two token streams, one for what is said and one for how it sounds, and predict them in that order to continue a recording.

AudioLM converts audio into semantic tokens from a self-supervised speech model and acoustic tokens from a neural codec, then models them with three decoder-only Transformers. Trained on 60,000 hours of untranscribed English speech, it continues a 3-second prompt in the speaker's voice, and in a listening test people told its continuations from real speech 51.2% of the time, close to the 50% of guessing.

The page builds both tokenizers (a residual quantizer you can step through, and k-means over a speech model's hidden layer), counts the sequence lengths that force the design into three stages, shows what each stage conditions on, and reads the listening test with its p-value worked out.

Why raw audio is a hard sequence to model

A language model like GPT-3 generates text one token at a time, each token sampled from a distribution conditioned on everything before it. Audio at 16 kHz has 16,000 samples per second, so a 10-second clip is a sequence of 160,000 numbers. Self-attention compares every position with every other, and its cost grows with the square of the length: 160,000 positions means 2.6 × 10¹⁰ pairs per layer, against about 10⁶ for the 1,000-token sequences the paper calls comfortable for a standard Transformer.

Earlier work reached the problem from two sides. WaveNet (2016) models raw samples autoregressively and sounds natural over short spans, but without text or other conditioning it produces babbling: speech-like sounds with no words. GSLM (Lakhotia et al., 2021) clusters the features of a self-supervised speech model into a few hundred discrete units, removes repeated units, and trains a Transformer on them; its output contains real words and sentences, but its unit-to-speech synthesizer was trained on one speaker, so every output has that voice. AudioLM's goal is both properties at once: long-range structure (words, syntax, melody) and the acoustic detail of any speaker or recording.

Tokenize, model, detokenize

AudioLM has three parts. A tokenizer maps a mono waveform $x \in \mathbb{R}^{T}$ to a sequence $h = (h_1, \dots, h_{T'})$ of discrete tokens from a finite vocabulary, with $T' \ll T$. A decoder-only Transformer, the same architecture as GPT, is trained to maximize the likelihood of those token sequences:

At inference the Transformer samples tokens one by one, and a detokenizer maps the sampled tokens back to a waveform $\hat{x}$. The tokenizer and detokenizer are trained first and then frozen; the language model never backpropagates into them. The paper gives a simpler training setup as the reason: the tokenizers are fixed functions, and the language model's inputs and targets are plain integers.

AudioLM uses two tokenizers whose tokens hold different information: acoustic tokens from a neural codec and semantic tokens from a self-supervised speech model, described in the next two sections.

Acoustic tokens: SoundStream and residual vector quantization

SoundStream is a neural audio codec from the same group (Zeghidour et al., 2021): an encoder network compresses audio into discrete codes and a decoder network reconstructs the audio from them, trained with reconstruction and adversarial losses so the output sounds like the input. Its convolutional encoder has four downsampling blocks with strides 2, 4, 5 and 8, so it outputs one embedding vector per $2 \cdot 4 \cdot 5 \cdot 8 = 320$ input samples. At 16 kHz that is 50 vectors per second, one per 20 ms frame.

Each 20 ms vector has to become discrete. The simplest option is a single codebook: a learned table of vectors, with the frame replaced by the index of its nearest entry. AudioLM's codec spends 120 bits per frame (6,000 bits per second at 50 frames per second), and a single codebook with 120 bits of index would need $2^{120}$ entries. Residual vector quantization (RVQ) gets the same bit budget from a stack of small codebooks. The first codebook quantizes the frame; the second quantizes the error the first one left; each later level quantizes the error left by all the levels before it:

In (2), $y$ is the encoder output for one frame, $Q_i$ returns the entry of codebook $i$ nearest to its input, and the token for level $i$ is that entry's index. AudioLM uses $Q = 12$ levels of $N = 1024$ entries each, so every frame becomes 12 integers of 10 bits, and the codec stores $12 \times 1024 = 12{,}288$ vectors instead of $2^{120}$. Making change with coins works the same way: the largest coin gets close, smaller coins fix the remainder. The analogy breaks in two places: codebook entries are learned vectors, not fixed denominations, and greedy nearest-entry choice at each level is not guaranteed to find the best combination.

In the figure the error drops by roughly half per level, because each toy codebook has half the radius of the one before. The tokens are ordered by importance: level 1 alone fixes the coarse position, and level 8 moves the reconstruction by under 0.01. SoundStream's learned codebooks behave the same way, and the paper uses that ordering. The first few levels recover what the paper calls coarse acoustic properties (speaker identity, recording conditions); the later levels add fine detail. Table I of the paper puts numbers on it: SoundStream at 4 levels (2,000 bps) reconstructs speech with a ViSQOL score of 3.3, and at 12 levels (6,000 bps) with 3.9. ViSQOL is an automatic estimate of how similar a reconstruction sounds to the reference, on a scale of 1 to 5.

A 10-second clip becomes a matrix $Y$ of $500 \times 12$ tokens. To feed it to a Transformer, AudioLM flattens it row by row, frame 1 levels 1 to 12, then frame 2 levels 1 to 12, and so on, into 6,000 tokens. Index 517 means different things at level 1 and level 3, so each token gets an offset by level, giving every level its own range of ids:

where $i$ is the position in the flattened sequence. A level-3 token with index 517 becomes $517 + 2 \cdot 1024 = 2565$, and the model's vocabulary for 12 levels has 12,288 ids.

Semantic tokens: k-means on w2v-BERT

The second tokenizer starts from w2v-BERT (Chung et al., 2021), a self-supervised speech model. Like wav2vec 2.0 it learns from untranscribed audio by hiding parts of the input and predicting them, and like BERT it adds a masked-prediction loss over discrete targets; the training never sees text. The XL version AudioLM uses has 0.6B parameters: a convolutional front end that reduces the frame rate 4-fold, 12 Conformer blocks trained with the contrastive loss, and 12 more trained with the masked-prediction loss. A Conformer block is a Transformer layer with an added convolution module, which helps it model local patterns in the audio. Its hidden vectors come out at 25 per second, one per 40 ms, each 1,024-dimensional.

AudioLM takes the vectors from the 7th layer of the masked-prediction module (the 19th of the 24 blocks), standardizes each dimension to zero mean and unit variance, and runs k-means with $K = 1024$ clusters: k-means places 1,024 centroid vectors so that every training vector lies close to one of them. A semantic token is the index of the nearest centroid. One second of speech becomes 25 tokens of 10 bits, 250 bits per second. The standardization step, which the paper reports “significantly improves” how well the tokens separate phonemes, keeps a few high-variance dimensions from dominating the k-means distances.

The layer and $K$ were chosen by measuring several candidates. Phonetic discriminability is scored with ABX: take three recordings A, B and X, where A and X are the same three-phoneme word (say “bit”) and B differs in the middle phoneme (“bet”), and count how often X is closer to B than to A in the representation. Within-speaker ABX takes all three from one speaker; across-speaker takes X from a different speaker than A and B. The paper also used two language-modeling probes described in the results section, plus listening to a few continuations. Layer 7 with 1,024 clusters came out best.

Why one tokenizer is not enough

The two token types fail in opposite directions. Table I of the paper compares them on ABX error (lower means the tokens separate phonemes better) and on ViSQOL after training a SoundStream decoder to reconstruct audio from each:

Semantic tokens at 250 bps make fewer than half as many ABX errors as acoustic tokens at 6,000 bps, and reconstruct speech so badly (ViSQOL 1.1, the bottom of the scale) that more bits barely help: the 6,000 bps version still scores 1.4. The w2v-BERT objective rewards predicting masked content, so the features keep what identifies phonemes and words and are not trained to keep detail such as the exact voice and room, which the decoder then cannot recover. SoundStream is trained to reproduce the waveform, and none of its losses depends on phoneme identity, so nothing in training makes phonemes easy to read off the tokens, which the ABX errors of 17.8 to 28.7 reflect.

The paper then trains a Transformer on acoustic tokens alone and prompts it with 4 seconds of speech. The continuations keep the speaker and the recording conditions and lose the language: the words are inconsistent and often babbling, like WaveNet's. A score and a recording of the same piece divide the information in a similar way. The score fixes the notes and timing and leaves out the instrument and the hall; the recording has everything but makes the notes hard to read off. The analogy overstates the separation: semantic tokens also carry much of the rhythm and intonation of speech, as the results section shows, and they are learned clusters, not symbols someone designed.

How AudioLM works: semantic first, then coarse, then fine

AudioLM models both token types, semantic first. It assumes the next semantic token depends on past semantic tokens and not on past acoustic ones:

where $z$ are the semantic tokens and $y$ the acoustic tokens. Under (4), the semantic sequence can be generated for the entire clip without looking at acoustics, and the acoustic tokens can then be generated conditioned on it. The alternative, one model over an interleaved sequence of both token types, would have to process every token in a single context, and Figure 2 counts what that costs.

At the default 10 seconds, raw samples would be 160,000 positions, and one model over semantic plus all 12 acoustic levels would read 6,250. AudioLM's three stages read 250 semantic tokens (stage 1); those 250 as a prefix plus 2,000 coarse tokens (stage 2, 2,250 in all); and four chunks of at most 1,800 coarse-plus-fine tokens (stage 3). Their summed attention cost is about 40% of the attention cost of one model over all 6,250 tokens. The saving grows with clip length (the readout's multiplier rises), because stage 3's chunks stay the same size. Below about 4 seconds the three stages cost slightly more than one model, since a single chunk then holds every acoustic token of the clip. The paper trains on crops of 30 seconds for stage 1 (750 semantic tokens), 10 seconds for stage 2 and 3 seconds for stage 3.

Each stage is a separate decoder-only Transformer, trained with next-token cross-entropy on the true tokens of its own sequence (teacher forcing). Stage 1 models semantic tokens alone:

Stage 2 predicts the acoustic tokens of the first $Q' = 4$ levels, the coarse ones, conditioned on the semantic tokens of the whole clip:

Its input sequence is all the semantic tokens first, then the coarse tokens flattened frame by frame: $(z_1, \dots, z_{T_S}, y_1^1, \dots, y_1^{4}, y_2^1, \dots)$. The loss covers only the acoustic positions; the paper names $y_1^1$ as the first token predicted in training. Because the whole semantic sequence comes first, a coarse token at time $t$ sees semantic tokens from later in the clip, while the causal mask (each position attends only to positions before it) still hides later acoustic tokens. In the second stage, two acoustic frames correspond to each semantic token (50 Hz against 25 Hz), so there are $2Q' = 8$ coarse tokens per semantic token before repeats are removed.

Stage 3 predicts the 8 fine levels from the coarse ones, without semantic tokens:

It conditions on all the coarse tokens of its chunk, the fine tokens of earlier frames, and the lower levels of the current frame. Dropping the semantic tokens rests on a second assumption, that fine detail is conditionally independent of content once the coarse acoustics are known. A further assumption, that fine detail is determined locally, lets stage 3 run on non-overlapping 3-second chunks, so its sequence length stays at 1,800 tokens whatever the length of the clip.

Click a coarse token at frame 1 and stage 2 conditions on all three semantic tokens, including those for frames 3 to 6. Click a fine token and the semantic row goes faint, while all 24 coarse tokens of the chunk turn teal, including those of frames after the target. The strip shows the same sets as positions: each stage is an ordinary causal language model, and its conditioning set in (5), (6) or (7) is whatever the sequence places before the target.

All three Transformers have the same shape: 12 layers, 16 attention heads, width 1,024, feed-forward width 4,096, dropout 0.1, and the relative position embeddings of T5, which add a learned bias to each attention score based on the distance between the two positions. Each stage trains on 16 TPUv4 chips with batch size 256 for 1M steps. The paper states 0.3B parameters per stage; the stated shape works out to about half that, a discrepancy discussed in the provenance notes below.

Training and sampling in code

One training example for stage 2, with tensor shapes for a 10-second crop. The paper says only that tokens are offset “properly”; putting semantic ids at 0 to 1,023 and the four coarse levels after them is one consistent choice, giving a vocabulary of 5,120.

# One stage-2 training example (speech model). Everything except
# stage2 is pretrained and frozen: w2v-BERT XL, the k-means, SoundStream.
x = random_crop(librilight_audio, seconds=10)   # (160000,) at 16 kHz
e = w2v_bert_xl(x, mlm_layer=7)                 # (250, 1024), 25 frames/s
z = kmeans_1024.nearest(standardize(e))         # (250,) ids in [0, 1024)
z = drop_consecutive_repeats(z)                 # e.g. 7 7 7 3 3 9 -> 7 3 9
Y = soundstream.encode(x)                       # (500, 12) ids in [0, 1024)
y = (Y[:, :4] + 1024 * arange(4)).reshape(-1)   # (2000,) row-major, offsets
seq = concat(z, 1024 + y)                       # id layout is ours (see text)
logits = stage2(seq[:-1])                       # causal; 12 layers, d = 1024
loss = cross_entropy(logits[len(z) - 1:], seq[len(z):])  # acoustic targets only
loss.backward()                                 # updates stage2 only

Stages 1 and 3 have the same structure. Stage 1 trains on 30-second crops of semantic tokens, with the loss on every position. Stage 3 trains on 3-second crops: 150 frames, the 600 coarse tokens as a prefix, and the 1,200 fine tokens as targets. The removal of repeated semantic tokens, borrowed from GSLM, applies in stages 1 and 2; a run like 7 7 7 3 3 9 becomes 7 3 9, which shortens the sequence. Durations then come from stage 2, whose acoustic frames keep the original timing.

Sampling uses temperature: the model's unnormalized scores (logits) $\ell_i$ become probabilities

and lower $\tau$ concentrates the probability on the top candidates. For logits (2, 1, 0) the probabilities are (0.665, 0.245, 0.090) at $\tau = 1$, (0.731, 0.209, 0.060) at 0.8, and (0.817, 0.154, 0.029) at 0.6. AudioLM uses 0.6 for stage 1, 0.8 for stage 2 and 0.6 for stage 3, chosen by the authors as a trade-off between diversity and semantic consistency. Continuation then runs the three models in order:

# Continue a 3 s prompt by 7 s (temperatures from Sec. IV-B).
z_p = drop_consecutive_repeats(semantic(prompt))     # <= 75 ids
Y_p = soundstream.encode(prompt)                     # (150, 12)
z = sample(stage1, prefix=z_p, temperature=0.6)      # semantic, up to 10 s
c = sample(stage2, prefix=concat(z, flat(Y_p[:, :4])),
           n_new=7 * 50 * 4, temperature=0.8)        # 1400 coarse ids
Y_c = concat(Y_p[:, :4], unflat(c, levels=4))        # (500, 4)
new = Y_c[150:]                                      # the 350 new frames
fine = [sample(stage3, prefix=flat(ch), n_new=len(ch) * 8, temperature=0.6)
        for ch in split(new, frames=150)]            # chunks: 150, 150, 50
audio = soundstream.decode(concat(Y_p, stack_levels(new, fine)))  # (160000,)

Continuing a 3-second prompt

To continue a prompt, AudioLM tokenizes it with both tokenizers. A 3-second prompt gives up to 75 semantic tokens and a $150 \times 12$ matrix of acoustic tokens. Stage 1 extends the semantic sequence. Stage 2 reads the full semantic sequence, prompt plus continuation, followed by the prompt's coarse tokens, and samples coarse tokens for the new frames. Stage 3 fills in the fine levels, and the SoundStream decoder converts all 12 levels, prompt and continuation, into a waveform.

The prompt's coarse acoustic tokens carry the voice into the continuation. Without them stage 2 would see only content and would pick a voice at random; with them, its context holds 600 coarse tokens of one speaker in one room immediately before the first token it samples, and the sampled tokens continue that voice. The paper also runs AudioLM without any prompt, sampling the semantic tokens from scratch, which produces speech with varying speakers and content, and in an “acoustic generation” mode, which takes the true semantic tokens of a test utterance and samples only the acoustics.

What the semantic and acoustic tokens carry

All speech components (SoundStream, w2v-BERT, the k-means, the three Transformers) are trained on unlab-60k, the 60,000-hour unlabeled part of Libri-Light, which is English audiobook speech with mixed recording quality. Earlier textless models used its 6,000-hour clean subset. Evaluation uses LibriSpeech test-clean, whose speakers are excluded from Libri-Light.

Content is in the semantic tokens. In acoustic generation, stages 2 and 3 resynthesize test utterances of 4 to 10 seconds from their true semantic tokens, three times each, and an automatic speech recognition (ASR) system, a Conformer Transducer-L, transcribes the result. Word error rate is 6.0% (character error rate 3.4%), against 2.5% (0.8%) for the original audio and 2.6% (0.9%) for the audio passed through SoundStream. Since the codec alone adds almost nothing, most of the extra errors come from mapping semantic to acoustic tokens; the paper names proper nouns as the main source, followed by sentences ending at the wrong place and generated background noise. GSLM's resynthesis of the same utterances gets 6.6% WER, always in the same single voice.

Speaker identity is in the acoustic tokens. A convolutional speaker classifier trained on 291 LibriSpeech speakers identifies 100% of SoundStream reconstructions. On acoustic generations from the true semantic tokens it names the original speaker 3.2% of the time, above the 0.34% of guessing at random. On 7-second continuations of 3-second prompts (classifier run on the continuation only) it names the prompt's speaker 92.6% of the time. By listening, the authors report that rhythm and intonation vary only slightly between acoustic generations from the same semantic tokens, so prosody sits mostly in the semantic tokens.

Stage 1 learns words and some grammar. Two zero-shot probes from the ZeroResource Challenge 2021 compare the probability a model assigns to two utterances, normalized by length. sWUGGY pairs a word with a similar non-word (“brick” and “blick”); sBLIMP pairs a grammatical sentence with an ungrammatical one (“the dogs sleep” and “the dog sleep”). A model scores a point when it prefers the real word or the grammatical sentence, so 50 is chance.

AudioLM has the highest score of the models without text supervision on all three columns, including the non-causal ones that see the full utterance in both directions. Its sBLIMP of 64.7 is 8% higher, relative, than CPC-BERT's 59.9, and above the forced-alignment topline trained on phonetic transcriptions. Without length normalization the sBLIMP score is 67.5, above the phone topline as well; the paper reports the normalized number because ungrammatical sBLIMP sentences are on average longer, and an unnormalized likelihood favors shorter sequences.

Can people tell? The listening test and the detector

Ten raters screened for English proficiency rated 100 clips of exactly 10 seconds, 1,000 ratings in all. Half were real LibriSpeech utterances passed through SoundStream, so codec artifacts could not give the answer away; half were a real 3-second prompt followed by a 7-second AudioLM continuation. Raters knew the first 3 seconds were always real and labeled each clip as original or synthetic. Of the 1,000 labels, 51.2% were correct, and the paper reports that this is not significantly different from guessing, with $p = 0.23$.

With 1,000 independent coin flips, the probability of 512 or more heads is 0.234, so the reported p-value is a one-sided test; the two-sided value is 0.467. A one-sided test needs 527 correct labels (52.7%) to reach p = 0.05. The labels are also not independent: each clip was rated by 10 people, and if raters tend to agree on a clip, the effective sample size is closer to 100 than to 1,000. The “100 independent clips” setting shows that extreme, where chance accuracy spreads over roughly 40% to 60% and the 5% line moves out to 59% (61% two-sided). Under every variant, 51.2% stays well inside the range of guessing, as the paper concludes. The test also covers only 7-second continuations, read speech, and one dataset.

A classifier separates AudioLM continuations from real speech easily. The authors trained the same convolutional architecture as the speaker classifier, on 1-second crops from LibriSpeech train-clean-100, to separate AudioLM continuations from SoundStream-compressed real speech. On a balanced test set it is right 98.6% of the time. The real side of the training set is compressed too because, against raw speech, the task is trivial (the paper reports quick convergence to 100%) and codec artifacts would confound what the classifier learns.

Piano

The same framework, retrained on 40,000 hours of internal piano recordings from beginner to expert level, continues piano music. Here the codec uses 3 quantizer levels with codebooks of $2^{14} = 16{,}384$ entries, enough for high reconstruction quality, so stage 2 predicts all three levels and stage 3 is dropped. Prompts are 4 seconds from the MAESTRO dataset. In a test with 10 raters and 15 pairs of 20-second continuations from the same prompt, one from AudioLM and one from a model trained only on acoustic tokens, raters preferred AudioLM in 83.3% of pairs. The authors describe both as equally high in audio quality and only AudioLM's as keeping consistent melody and rhythm. There is no symbolic music (MIDI or notes) anywhere in training; the semantic tokens are k-means clusters of a w2v-BERT model trained on the piano audio.

Limitations

Generation is sequential. Every token costs one forward pass, and a 7-second continuation samples about 4,400 tokens, of which the stage-3 chunks can run in parallel; the paper gives no timing. The detector is trained against AudioLM itself, so its 98.6% says nothing about other generators.

The w2v-BERT layer and $K$ were chosen partly by listening to continuations, and the paper gives no measurement behind the three sampling temperatures. The speech model is trained on English audiobooks, and the paper notes that continuations may not keep the accent or dialect of speakers underrepresented in that data. The piano data is internal, so the piano result cannot be reproduced. The paper links audio samples; no official code or weights were released.

The paper lists the misuse risk directly: continuing 3 seconds of someone's voice can be used to spoof voice-based authentication or impersonate a speaker, and the detector is offered as a partial mitigation.