VerifiedarXiv:1609.0349928 min
Audio · Generative models

WaveNet: A Generative Model for Raw Audio

A convolutional network generates audio one sample at a time, choosing each of 16,000 samples per second from 256 values.

WaveNet treats a waveform the way a language model treats text: a sequence where each element is predicted from all the ones before it. Stacked convolutions with doubling gaps let each prediction see thousands of past samples, and with text as an extra input its speech was rated more natural than Google's existing synthesizers.

Explaining the paperWaveNet: A Generative Model for Raw Audiovan den Oord, Dieleman, Zen, Simonyan, Vinyals, Graves, Kalchbrenner, Senior, Kavukcuoglu · arXiv 2016 (SSW9) · arXiv:1609.03499 ↗

A September 2016 DeepMind paper that dropped the vocoder from speech synthesis and generated the waveform directly, sample by sample, with a probability model borrowed from image generation.

Before WaveNet, text-to-speech systems produced audio in one of two ways. Concatenative systems stitched together short pieces of recorded speech from a large database. Parametric systems predicted a compact description of the sound every 5 ms (a spectral envelope, a pitch value) and passed it to a vocoder, a signal-processing program that turns those parameters back into a waveform. The first sounded natural but needed a large database and could not easily change voice; the second was compact and flexible but sounded muffled. Aäron van den Oord and eight co-authors at DeepMind and Google asked whether a neural network could skip the parameters and output the waveform itself, one amplitude value at a time.

The model is an adaptation of PixelCNN, which the same group had used to generate images pixel by pixel earlier in 2016. A few ideas explain it:

  1. the probability of a waveform factorizes into one prediction per sample, equation (1);
  2. causal convolutions make those predictions without looking ahead, and dilation lets a short stack see far back;
  3. each prediction is a softmax over 256 values, made possible by mu-law companding;
  4. a gated, residual layer repeated 30 or more times, equation (2);
  5. conditioning on a speaker or on text, equation (3);
  6. sampling one value at a time, and what that costs;
  7. the results: listening tests in US English and Mandarin, a multi-speaker model, music, and phoneme recognition.

Audio as a sequence of numbers

A digital recording measures air pressure at a fixed rate. The speech in this paper is sampled 16,000 times a second (16 kHz), and each measurement is stored as a 16-bit integer, one of 65,536 values. One second of speech is a list of 16,000 integers. A word lasts a few hundred milliseconds, so it spans several thousand samples, and the pitch of a voice shows up as a pattern that repeats every 40 to 200 samples or so (a pitch between 80 and 400 Hz).

PixelRNN, a recurrent image model from the paper that introduced PixelCNN, had modeled 64x64 images, 4,096 pixels. A two-second sentence is 32,000 positions, and structure exists at every scale in between: single samples within a pitch period, periods within a phoneme, phonemes within a word.

Predicting the next sample

Any probability distribution over a sequence x={x1,…,xT}\mathbf{x} = \{x_1, \ldots, x_T\} can be written, by the chain rule of probability, as a product of one conditional distribution per element:

p(x)=∏t=1Tp(xt∣x1,…,xt−1)p(\mathbf{x}) = \prod_{t=1}^{T} p(x_t \mid x_1, \ldots, x_{t-1})
(1)

The equation makes no approximation. It converts the problem of assigning a probability to a whole waveform into 16,000 problems per second of the same form: given everything so far, what is the distribution of the next value? A single network answers all of them. This is the same factorization a GPT-style language model uses for text, with audio samples in place of tokens.

Training maximizes the log of (1), which is a sum: log⁡p(x)=∑tlog⁡p(xt∣x<t)\log p(\mathbf{x}) = \sum_t \log p(x_t \mid x_{<t}). For a concrete step, suppose the network's output at time 4,800 assigns probability 0.31 to the value that the recording actually has at time 4,801. That position contributes −ln⁡0.31=1.17-\ln 0.31 = 1.17 nats (the unit of a natural-log loss) to the loss. A network that knew nothing and spread its probability evenly over 256 values would pay ln⁡256=5.55\ln 256 = 5.55 nats at every position, so 5.55 nats is the baseline, and the distance below it measures what the network has learned. Because the log-likelihood is computed exactly, the authors could tune hyper-parameters on held-out data and see directly whether a model was overfitting.

During training the whole recording is known, so every conditional in (1) can be evaluated at once: the input is the true waveform, and the target at each position is the true next sample. This is called teacher forcing. The loop below is one training step. Nothing in it is sequential; the only constraint is that output tt must not depend on any input after tt, which the next section builds in.

# One training step on a batch of 4 one-second clips at 16 kHz
audio  = load_clips()                    # 4 x 16000 floats in (-1, 1)
codes  = mu_law_encode(audio)            # 4 x 16000 ints in 0..255
x      = one_hot(codes[:, :-1], 256)     # inputs: samples 1..15999
target = codes[:, 1:]                    # targets: samples 2..16000 (shifted by one)
logits = wavenet(x)                      # 4 x 15999 x 256; output t sees inputs <= t
loss   = cross_entropy(logits, target)   # mean of -log p(true code), in nats
loss.backward()                          # one pass gives gradients for 63,996 targets
optimizer.step()

How WaveNet works: causal convolutions

A one-dimensional convolution with filter width 2 computes each output from two adjacent inputs. In a standard (centered) convolution, the output at tt might read xtx_t and xt+1x_{t+1}, which is cheating for a model that has to predict xt+1x_{t+1}. A causal convolution reads only xt−1x_{t-1} and xtx_t. The paper implements this by running an ordinary convolution and shifting its output a few steps later in time, equivalent to padding the input on the left. Stack several causal layers and no path from the input to output tt ever passes through a later time, so the prediction emitted at tt, which is the distribution for xt+1x_{t+1}, depends only on x1,…,xtx_1, \ldots, x_t.

The receptive field of an output is the set of inputs it can depend on. For width-2 causal layers without gaps it grows by one sample per layer: the paper's Figure 2 has four layers and a receptive field of 5, which it gives as layers plus filter length minus one. Reaching 240 ms of audio at 16 kHz, 3,840 samples, this way would take 3,839 layers.

A dilated convolution skips inputs: with dilation dd, a width-2 filter reads xt−dx_{t-d} and xtx_t instead of two neighbours. It is equivalent to a wider filter with zeros between its taps, but computes only the two nonzero taps. WaveNet doubles the dilation at every layer, 1, 2, 4, 8, and so on. With dilations d1,…,dLd_1, \ldots, d_L and filter width kk, the receptive field is

RF=(k−1)∑l=1Ldl+1\text{RF} = (k-1)\sum_{l=1}^{L} d_l + 1

which for doubling dilations and k=2k = 2 is 2L2^L: four layers see 16 samples instead of 5. Figure 1 lets you check this on the paper's own diagrams. Each dot is one value at one time; each layer reads the layer below at two times. Click a dot on the top row, or move the output-time slider, to light up every value that output depends on, and switch dilation off to get the paper's Figure 2.

Figure 1 · dilated causal convolutions
4
40
Width-2 causal layers over 40 input samples. The teal lines are the paths into the chosen output, the amber dots the inputs it reads. With dilation on, each layer doubles the gap and the receptive field doubles; with it off, each layer adds one sample. Pick an early output to see the cone cut off by the start of the recording, as happens to the first outputs of every training clip.

Inputs to the right of the selected output stay dark at every setting: no output reads a later sample, whatever the depth or dilation. With five dilated layers the cone covers 32 inputs with each one reached by exactly one path, so the stack computes a nonlinear function of those 32 samples using 5 layers of two-tap filters.

Dilated convolutions and the receptive field

WaveNet doubles the dilation up to 512 and then starts again from 1:

1,2,4,…,512,  1,2,4,…,512,  1,2,4,…,5121, 2, 4, \ldots, 512,\; 1, 2, 4, \ldots, 512,\; 1, 2, 4, \ldots, 5121,2,4,…,512,1,2,4,…,512,1,2,4,…,512\begin{gathered} 1, 2, 4, \ldots, 512, \\ 1, 2, 4, \ldots, 512, \\ 1, 2, 4, \ldots, 512 \end{gathered}

One block of ten layers (dilations 1 to 512) has a receptive field of 1+1023=10241 + 1023 = 1024 samples, which the paper describes as a nonlinear, more discriminative counterpart of a single convolution 1,024 samples wide. Each additional block adds another 1,023 samples and ten more layers of computation. Three blocks reach 3,070 samples, 192 ms at 16 kHz.

The paper gives two reasons for the pattern: exponential growth of the receptive field within a block, and more model capacity from stacking blocks. It does not explain the cap at 512. Figure 2 shows the alternative of never resetting: pure doubling across 30 layers would reach 2302^{30} samples, about 18.6 hours, and only the first ten layers would combine samples less than 1,024 apart (64 ms). With the reset, each block of ten layers works again at every gap from 1 to 512 samples.

Figure 2 · receptive field against depth
30
Receptive field at 16 kHz on a log scale for width-2 causal layers. No dilation grows by one sample per layer; doubling with a reset every 10 layers (the paper's pattern) grows by 1,023 samples per block; doubling without reset leaves the chart at 28 layers. Drag the slider to read exact sample counts.

The shaded band marks what the paper reports for its trained models: 240 ms for the text-to-speech networks and about 300 ms for the multi-speaker network. The teal curve enters that band at 40 layers (4,093 samples, 256 ms). The paper does not say how many layers its models had; four or five blocks of the example pattern, or larger filters, would cover the reported values.

Section 2.6 of the paper adds another way to extend context: a separate, smaller context stack processes a long stretch of audio, possibly with pooling so it runs at a lower rate, and conditions the main WaveNet, which processes only the most recent stretch. The paper does not report which experiments used one.

Why a softmax over 256 values

The network has to output a probability distribution for the next sample. The obvious choice for a continuous quantity is a Gaussian, or a mixture of a few: the network outputs a mean and a variance, and training minimizes the negative log-likelihood. WaveNet instead treats the next sample as one of a fixed set of discrete values and outputs a softmax over them, a list of probabilities that sum to 1. PixelRNN had found that a softmax over the 256 pixel intensities beat continuous mixtures for images, even though intensities are naturally ordered numbers, because a categorical distribution assumes nothing about the shape: it can be skewed, flat, or have several peaks.

Figure 3 makes that argument with an invented distribution. The amber bars put 60% of their probability on one value region and 40% on another. A softmax can match them exactly. The teal curve is the best single Gaussian for those bars, the one with the same mean and variance.

Figure 3 · a softmax against a Gaussian
60 codes
An illustrative next-sample distribution with two bumps, and the maximum-likelihood Gaussian for it, both over the 256 codes. The readouts are the expected loss per sample in nats for each model; a uniform guess costs 5.55. Drag the separation to zero and the two models agree.

At zero separation the two models cost the same, 3.03 nats. At 60 codes apart the Gaussian costs 4.81 nats against the softmax's 3.70, and past about 135 codes apart the Gaussian is worse than guessing uniformly. Its mean stays at code 128, where the bars have almost no probability. A network trained with squared error, the regression alternative, would output that mean; sampling from such a model would produce values the data almost never contains.

The softmax has one output per possible value. With 16-bit audio that is 65,536 outputs at every one of the 16,000 positions per second. To make it tractable, WaveNet first reduces the number of values to 256.

Mu-law companding: from 65,536 values to 256

The simplest reduction to 256 values is linear 8-bit quantization: divide the range from -1 to 1 into 256 equal steps, each 0.0078 wide. Speech is quiet much of the time, though, and a quiet sound that swings by only a few hundredths gets only a handful of steps, so its quantization error is large compared with the signal. Telephone systems solved this decades ago with mu-law companding (compress, then expand): apply a logarithmic curve before quantizing, so that small amplitudes get fine steps and large ones get coarse steps. WaveNet uses the continuous mu-law curve,

f(xt)=sign⁡(xt) ln⁡(1+μ∣xt∣)ln⁡(1+μ)f(x_t) = \operatorname{sign}(x_t)\,\frac{\ln(1 + \mu \lvert x_t \rvert)}{\ln(1 + \mu)}

with −1<xt<1-1 < x_t < 1 and μ=255\mu = 255, and then divides the output range evenly into 256 values. The curve's slope is μ/ln⁡(1+μ)≈46\mu / \ln(1 + \mu) \approx 46 at zero and about 0.18 at full scale, a ratio of 1+μ=2561 + \mu = 256. Mapped back to amplitude, the 256 steps range from 0.00017 wide near silence (about 6 steps of 16-bit audio) to 0.043 wide near full scale (about 1,400). The network predicts the code of the next sample, an integer from 0 to 255; turning a code back into an amplitude applies the inverse curve.

Figure 4 measures the result on a sine wave. The top panel plots signal-to-noise ratio (signal power over the power of the quantization error, in dB) against the sine's peak level in dBFS, decibels relative to the largest value the format can store. Lower the level slider and watch linear 8-bit fall off while mu-law holds; move mu toward 0 to turn mu-law back into linear quantization.

Figure 4 · linear against mu-law quantization
-30 dBFS
255
8-bit linear and 8-bit mu-law quantization of a sine wave, with 16-bit linear dashed for reference. Below, a stretch of the sine at the chosen level (white) with both reconstructions. The quantizer rounds to 256 evenly spaced levels of the compressed value.

With μ=255\mu = 255, the mu-law signal-to-noise ratio stays between 35 and 39 dB from -40 dBFS up to full scale and is still 29 dB at -50 dBFS. Linear 8-bit loses 6 dB for every halving of the level: 30 dB at -20 dBFS, 11 dB at -40, and 0 dB from -48 down, where every sample of the sine rounds to zero. At the loudest levels linear 8-bit is better (49 dB against 38 at -1 dBFS); mu-law gives up resolution at the top to keep the error roughly proportional to the signal everywhere. The paper reports that, especially for speech, the 8-bit mu-law reconstruction sounded very similar to the original. Its listening test puts a number on it: natural English recordings scored 4.55 out of 5 in 16-bit and 4.46 after 8-bit mu-law encoding.

The gated residual layer

Every layer of the stack applies the same unit, taken from the gated PixelCNN:

z=tanh⁡(Wf,k∗x)⊙σ(Wg,k∗x)\mathbf{z} = \tanh(W_{f,k} * \mathbf{x}) \odot \sigma(W_{g,k} * \mathbf{x})
(2)

Here ∗* is a dilated causal convolution, kk is the layer index, ⊙\odot multiplies elementwise, and σ\sigma is the sigmoid. The layer runs two convolutions on the same input. The filter branch tanh⁡(Wf,k∗x)\tanh(W_{f,k} * \mathbf{x}) produces values between -1 and 1; the gate branch σ(Wg,k∗x)\sigma(W_{g,k} * \mathbf{x}) produces values between 0 and 1 that scale each filter value. With 32 channels, a gate value of 0.9 on channel 5 at time tt lets 90% of that channel's filter output through, and 0.05 almost blocks it. The same multiplicative gate controls what an LSTM writes to its memory. The paper reports that this unit worked significantly better than a ReLU for audio in early experiments.

The output z\mathbf{z} then goes two ways (the paper's Figure 4). A 1x1 convolution, which mixes channels at each time step independently, produces a correction that is added to the layer's input: the residual connection of ResNet, which lets a stack of 30 or more layers train because each layer only has to learn a change to its input. A 1x1 convolution of z\mathbf{z} also goes straight to the output as a skip connection. The output head adds up the skip outputs of all layers, applies ReLU, a 1x1 convolution, ReLU, another 1x1 convolution down to 256 channels, and a softmax at every time step.

The paper gives no layer widths. With the sizes of the ibab/tensorflow-wavenet reimplementation (32 channels in the residual path and gates, 512 in the skip path) and the paper's 30-layer example, one second of audio flows through the network like this:

def residual_layer(x, d, W):             # x: batch x T x 32; W: this layer's weights
    f = causal_conv(x, W.f, dilation=d)  # width 2: reads x[t - d] and x[t]
    g = causal_conv(x, W.g, dilation=d)
    z = tanh(f) * sigmoid(g)             # equation (2), batch x T x 32
    skip = conv1x1(z, W.skip)            # batch x T x 512, sent to the output head
    return x + conv1x1(z, W.res), skip   # the residual sum feeds the next layer

def wavenet(x_onehot):                   # batch x T x 256
    h = causal_conv(x_onehot, W_in)      # width 2, 256 -> 32 channels
    skips = 0
    for k, d in enumerate([1, 2, 4, 8, 16, 32, 64, 128, 256, 512] * 3):
        h, s = residual_layer(h, d, W_layer[k])
        skips = skips + s
    out = conv1x1(relu(conv1x1(relu(skips), W_p1)), W_p2)
    return out                           # batch x T x 256 logits

Each layer has 22,112 parameters, weights and biases (4,160 in the two dilated convolutions, 1,056 in the residual 1x1, 16,896 in the skip 1x1), and the whole network about 1.07 million, of which 393,984 sit in the two 1x1 convolutions of the head. Figure 5 is the paper's Figure 4 with those numbers attached: select any node to see what it computes and its shape for one second of input.

Figure 5 · one residual layer and the output head
The residual layer (left, repeated 30 times) and the output head (right). Select a node, on the canvas or in the menu, to read its computation and tensor shape; the conditioning buttons add the input h of the next section. Channel counts are from the ibab reimplementation.

The skip path is the only route to the output: the residual path of the last layer ends at the top of the stack. Because every layer writes to the skip sum directly, the gradient of the loss reaches layer 1 through its own skip connection without passing through the 29 layers above it.

Conditioning on a speaker or on text

Equation (1) describes audio in general. To make speech of a particular speaker, or speech that says a particular sentence, the model takes an extra input h\mathbf{h} and every conditional depends on it:

p(x∣h)=∏t=1Tp(xt∣x1,…,xt−1,h)p(\mathbf{x} \mid \mathbf{h}) = \prod_{t=1}^{T} p(x_t \mid x_1, \ldots, x_{t-1}, \mathbf{h})
(3)

Global conditioning uses one vector for the whole clip, such as the identity of the speaker. It enters each layer as a learned projection added inside both halves of the gated unit:

z=tanh⁡(Wf,k∗x+Vf,kTh)⊙σ(Wg,k∗x+Vg,kTh)\mathbf{z} = \tanh(W_{f,k} * \mathbf{x} + V_{f,k}^{T}\mathbf{h}) \odot \sigma(W_{g,k} * \mathbf{x} + V_{g,k}^{T}\mathbf{h})z=tanh⁡(Wf,k∗x+Vf,kTh)⊙σ(Wg,k∗x+Vg,kTh)\begin{aligned} \mathbf{z} = & \tanh(W_{f,k} * \mathbf{x} + V_{f,k}^{T}\mathbf{h}) \\ &\qquad \odot \sigma(W_{g,k} * \mathbf{x} + V_{g,k}^{T}\mathbf{h}) \end{aligned}

With 109 speakers and a one-hot h\mathbf{h} (a vector of 109 entries, all 0 except a 1 at the speaker's index), Vf,kThV_{f,k}^{T}\mathbf{h} selects one row of the 109x32 matrix Vf,kV_{f,k}: a 32-dimensional bias, learned per speaker and per layer, added at all 16,000 time steps. The music experiments use the same mechanism with a binary vector of tags (genre, instruments) in place of the speaker.

Local conditioning uses a second time series ht\mathbf{h}_t, usually at a lower rate than the audio. For text-to-speech it is a vector of linguistic features every 5 ms: phone identity, syllable stress, the number of syllables in the word, the position of the syllable in the phrase, and the position within the current phone and its duration. At 16 kHz, 5 ms is 80 samples. A transposed convolutional network (a learned upsampler) maps h\mathbf{h} to a series y=f(h)\mathbf{y} = f(\mathbf{h}) with one vector per audio sample, and each layer adds a 1x1 convolution of it:

z=tanh⁡(Wf,k∗x+Vf,k∗y)⊙σ(Wg,k∗x+Vg,k∗y)\mathbf{z} = \tanh(W_{f,k} * \mathbf{x} + V_{f,k} * \mathbf{y}) \odot \sigma(W_{g,k} * \mathbf{x} + V_{g,k} * \mathbf{y})z=tanh⁡(Wf,k∗x+Vf,k∗y)⊙σ(Wg,k∗x+Vg,k∗y)\begin{aligned} \mathbf{z} = & \tanh(W_{f,k} * \mathbf{x} + V_{f,k} * \mathbf{y}) \\ &\qquad \odot \sigma(W_{g,k} * \mathbf{x} + V_{g,k} * \mathbf{y}) \end{aligned}

The simpler alternative, repeating each 5 ms feature vector 80 times, worked slightly worse in the paper's experiments. Either way the text decides what is said and when, and the autoregressive part fills in how it sounds sample by sample. The text-to-speech WaveNets were also given the logarithm of the fundamental frequency of the voice (log F0) as a second local input, predicted by a separate model; the results section explains why it mattered.

Generating audio one sample at a time

Training evaluates all conditionals in parallel because the true waveform is available. Generation has no waveform until the model makes one. The procedure is ancestral sampling: compute the distribution for the next sample, draw one code from it at random, decode it, append it to the waveform, and repeat with the longer waveform as input. Figure 6 runs that loop. The predictive distributions in it come from a two-tap linear predictor, not a trained WaveNet; that predictor is the Gaussian autoregressive model the paper's Appendix A gives as the basis of classical linear prediction, equation (6) with two coefficients, discretized onto the 256 codes.

Figure 6 · sampling, one value per step
160
Each step shows the distribution over the next code (zoomed to 41 codes around its peak), the drawn code, and the waveform it extends. The shaded window is the part of the past the next prediction reads. Play, step, or scrub; the path is fixed, so scrubbing back shows the same draws.

The waveform starts from silence and the oscillation builds up from the random draws. Each draw is a single code, but because it becomes input, an unusual draw changes every later distribution. Nothing in the loop can be parallelized across time: step 4,801 cannot start until step 4,800 has drawn its value.

The direct way to implement the loop reruns the network on the last receptive field of codes at every step:

# Naive sampling: every step reruns the network on the last 3,071 codes
codes = [128] * 3071                     # start from silence (code 128 decodes to ~0)
for t in range(16000):                   # one second at 16 kHz
    window = one_hot(codes[-3071:], 256) # receptive field: 3,070 for the 30 dilated layers + 1 for the width-2 input conv
    p = softmax(wavenet(window)[-1])     # only the last of the 3,071 outputs is new
    codes.append(sample(p))              # draw one code; it becomes the next input
audio = mu_law_decode(codes[3071:])      # 16,000 floats in (-1, 1)

Almost all of that work repeats the previous step. In the 30 dilated layers, the newest output depends on 48,121 layer outputs, and all but 30 of them (the ones at the current time, one per layer) were already computed for earlier samples. Fast WaveNet (Paine et al., 2016) keeps a queue of recent outputs for every layer, sized to that layer's dilation, so each step computes exactly one new value per layer: cost proportional to the number of layers LL rather than 2L2^L. That is the "cached" readout in Figure 6.

Even cached, one second of audio requires 16,000 sequential passes through 30 or more layers, each a small amount of work that cannot start until the previous pass has finished. The paper reports no generation speed. DeepMind's 2017 follow-up, Parallel WaveNet, describes the original as poorly suited to massively parallel hardware; its distilled model generates speech more than 20 times faster than real time, and DeepMind's announcement put it at more than 1,000 times faster than the original.

Results: text-to-speech listening tests

The text-to-speech experiments used the single-speaker databases behind Google's production voices: 24.6 hours of North American English and 34.8 hours of Mandarin Chinese, each read by a professional female speaker, at 16 kHz. The baselines were Google's own systems of each kind, an LSTM-based parametric synthesizer (Zen et al., 2016) and an HMM-driven concatenative one (Gonzalvo et al., 2016), built from the same data and the same linguistic features. Raters were paid native speakers, recruited by crowdsourcing; each item was rated by eight of them, and ratings given without headphones (about 40%) were discarded.

In the mean opinion score (MOS) test, each rater heard one sample and scored its naturalness from 1 (bad) to 5 (excellent). Table 1 of the paper:

speech samplesUS EnglishMandarin
LSTM-RNN parametric3.67 ± 0.0983.79 ± 0.084
HMM-driven concatenative3.86 ± 0.1373.47 ± 0.108
WaveNet (linguistic + log F0)4.21 ± 0.0814.08 ± 0.085
natural, 8-bit mu-law4.46 ± 0.0674.25 ± 0.082
natural, 16-bit linear PCM4.55 ± 0.0754.21 ± 0.071

In English, the best baseline (concatenative, 3.86) sat 0.69 below natural 16-bit speech and WaveNet (4.21) sat 0.34 below it, so WaveNet closed 51% of the gap. In Mandarin the best baseline was the parametric one (3.79), 0.42 below natural; WaveNet's gap was 0.13, a 69% reduction. Both WaveNet scores are above 4, which on the scale means "good", and the authors state that they were the highest MOS ever reported on those datasets and test sentences.

The paired comparison test played two systems saying the same sentence and asked which sounded more natural, with "no preference" allowed. Against the LSTM parametric system, English listeners preferred WaveNet with log F0 82.0% of the time and the LSTM 7.6%; against the concatenative system, 49.3% to 20.1%, with 30.6% expressing no preference.

The same test separates two WaveNets. WaveNet (L), conditioned on linguistic features only, produced natural sounds but sometimes stressed the wrong words. The authors attribute this to its 240 ms receptive field being too short for the F0 contour of a sentence, which rises and falls over seconds. WaveNet (L+F) also receives log F0 from a separate model that runs at 200 Hz, one value per 5 ms, so a few hundred steps of that model span a whole sentence. In English, listeners preferred L+F over L 37.9% to 17.8%. In Mandarin, a tonal language where pitch changes the meaning of a syllable, WaveNet (L) was not better than the LSTM baseline (23.3% against 25.0%, p = 0.476), while L+F beat it 29.3% to 12.5%. The abstract's claim of beating both baselines in both languages refers to the L+F model.

Speakers, music, and speech recognition

The multi-speaker experiment trained one WaveNet on VCTK, 44 hours from 109 English speakers, conditioned only on a one-hot speaker identity and not on text. With no text to follow, it produced speech-like babble: realistic phonemes and intonation forming words that do not exist. The paper attributes the lack of long-range coherence partly to the receptive field, about 300 ms, which covers the last two or three phonemes. One network modeled all 109 voices, and training on all of them gave better validation likelihood than training on one speaker. The samples also reproduced the room acoustics, recording quality, breathing, and mouth noises of each speaker.

For music, the authors trained on MagnaTagATune (about 200 hours of 29-second clips with tags from a set of 188) and on about 60 hours of solo piano from YouTube. Enlarging the receptive field was necessary for samples to sound musical, but even with several seconds of context the models drifted in genre, instrumentation, and volume from second to second. Conditioning on tags, as biases in the same way as the speaker identity, let the samples be steered toward a genre or instrument. The paper evaluates music only by listening and reports no numbers.

The last experiment uses WaveNet as a classifier. For TIMIT phoneme recognition, a mean-pooling layer after the dilated stack averages the activations into 10 ms frames (160 samples, a 160x reduction), followed by a few non-causal convolutions that classify each frame. Training with two losses, next-sample prediction and frame classification, generalized better than either alone and reached an 18.8% phone error rate on the test set, the best the authors knew of for a model trained on raw audio rather than on spectral features. Later raw-audio speech models such as wav2vec 2.0 also learn their own front end from the waveform.

Limits and later work

The paper's results come with constraints it states or that follow from its design. Generation is sequential, 16,000 dependent steps per second of audio. The receptive field of a few hundred milliseconds means long-range structure (prosody in speech, form in music) has to come from conditioning or not at all. Speech quality depended on external models for phone durations and log F0, so the system was still a pipeline with a front end, with WaveNet replacing the vocoder and acoustic model at the end. And the paper leaves out the configuration needed to reproduce it: layer counts, channel widths, training time, and sampling speed.

Follow-up work went after the first three constraints. Fast WaveNet (2016) cached layer outputs. Parallel WaveNet (2017) trained a feed-forward student network to match a trained WaveNet's distribution, which made generation fast enough for Google Assistant. Tacotron 2 (2017) replaced the linguistic features and F0 model with mel spectrograms predicted from characters by a sequence-to-sequence network and used a WaveNet as the vocoder, reaching a MOS of 4.53 against 4.58 for professional recordings. AudioLM (2022) kept the autoregressive idea but moved it from individual samples to discrete tokens: 25 per second for its semantic stream and 600 per second from a neural audio codec for its acoustic stream, instead of 16,000 samples, so a Transformer can model longer stretches of audio.

Provenance Verified against primary literatureHow we verify
van den Oord et al. (2016), arXiv 1609.03499v2Equations (1) to (3), the mu-law formula with mu = 255 and 256 levels, the global and local conditioning equations (unnumbered in the paper), the dilation pattern 1, 2, ..., 512 repeated, Table 1 (MOS), Table 2 (paired preferences), Appendix A equations (6) and (7), and Appendix B (16 kHz, 8-bit mu-law training data, 100 test sentences, eight raters per item, headphone filter).
Figure 4 of the paper (PDF page 4)The residual layer: dilated conv, tanh and sigma branches, elementwise product, one 1x1 convolution whose output goes both to the residual sum and to the skip path; the head sums the skips, then ReLU, 1x1, ReLU, 1x1, softmax.
ibab/tensorflow-wavenet (no official code exists)wavenet_params.json: filter_width 2, dilations 1..512 five times, residual and dilation channels 32, skip channels 512, quantization channels 256. wavenet/model.py, _create_dilation_layer: separate 1x1 "dense" (residual) and "skip" convolutions after tanh(filter) * sigmoid(gate); calculate_receptive_field = (filter_width - 1) * sum(dilations) + 1, plus filter_width - 1 for the input convolution. wavenet/ops.py, mu_law_encode: round((f + 1) / 2 * 255).
Gated PixelCNN, van den Oord et al. (2016b), arXiv 1606.05328The gated unit tanh(W_f * x) times sigmoid(W_g * x) that WaveNet reuses, and conditioning through added projections of a vector h.
Paine et al. (2016), Fast WaveNet, arXiv 1611.09482Caching each layer's past outputs cuts the cost of generating one sample from O(2^L) to O(L) for L layers.
van den Oord et al. (2017), Parallel WaveNet, arXiv 1711.10433; DeepMind blog, 22 Nov 2017The distilled feed-forward model generates speech "more than 20 times faster than real-time" (abstract) and is "more than 1000 times faster than the original" (blog). The original paper reports no generation speed.
Shen et al. (2017), Tacotron 2, arXiv 1712.05884A WaveNet conditioned on predicted mel spectrograms reaches MOS 4.53 against 4.58 for professional recordings.
ITU-T G.711; mu-law (Wikipedia)G.711 defines a discrete, segmented 8-bit mu-law code; the paper uses the continuous curve, which is the analog form of the same law.
This page's arithmeticReceptive fields: 1,024 per 10-layer block, 3,070 for three blocks (192 ms at 16 kHz), 5,116 for five (320 ms). Mu-law with mu = 255: slope 45.99 at 0 and 0.180 at 1 (ratio 1 + mu = 256); quantization steps from 0.000172 to 0.0427. Sine SNRs from our int8-level quantizer. 48,121 = the number of (layer, time) outputs one new sample depends on in the 30-layer stack. 1,073,760 parameters for 30 layers at the ibab channel sizes.
caveatFour scope notes; none changes the method. (1) The abstract says listeners rated WaveNet significantly more natural than the best parametric and concatenative systems "for both English and Mandarin". That holds for the WaveNet conditioned on linguistic features and log F0 (L+F), the only WaveNet in the MOS table. The WaveNet without F0 (L) was not better than the LSTM baseline in Mandarin: 23.3% against 25.0% preference, p = 0.476 (Table 2). (2) The paper reports receptive fields of 240 ms (TTS) and about 300 ms (VCTK) but not the layer counts, channel widths, or parameter counts behind them; its example pattern of 30 layers reaches only 3,070 samples, 192 ms at 16 kHz. The shapes on this page use a reimplementation's channel sizes. (3) Figure 4 draws one 1x1 convolution feeding both the residual and the skip path; common reimplementations use two separate 1x1 convolutions, which is what this page's parameter counts assume. (4) The paper cites ITU-T G.711 for mu-law but uses the continuous formula quantized uniformly to 256 levels, not G.711's segmented code. The paper gives no generation speed; the speed numbers here are from later DeepMind papers.

Questions you might still have

?

Is WaveNet a recurrent neural network?
No. It is a stack of one-dimensional convolutions with no recurrent connections, so during training all 16,000 outputs of a one-second clip are computed in one parallel pass. The only loop is at sampling time, when each drawn sample is fed back as the next input. The paper contrasts this with LSTMs, which it says need recurrence to build long contexts that dilated convolutions get more cheaply.

?

Why model the waveform instead of predicting a spectrogram or vocoder parameters?
Appendix A of the paper lists the problems of the parametric pipeline it replaced: vocoders introduce artifacts, and models of vocoder parameters average over possibilities and produce muffled, oversmoothed speech. Predicting samples directly avoids the vocoder. Later systems kept an intermediate representation for the text side: Tacotron 2 predicts mel spectrograms from characters and uses a WaveNet as the vocoder that turns them into samples.

?

Does training on 8-bit mu-law audio cap the quality?
Slightly. In the paper's listening tests, natural English recordings scored 4.55 in 16-bit PCM and 4.46 after 8-bit mu-law encoding, so the encoding alone cost 0.09 MOS points; WaveNet scored 4.21. In Mandarin the 8-bit recordings scored 4.25 and the 16-bit ones 4.21, a difference inside the error bars.

?

How does one network speak with 109 different voices?
The speaker identity enters every layer as a one-hot vector h, projected by learned matrices into one bias vector per layer for the filter and one for the gate, added at every timestep. Changing h changes those biases and nothing else. The paper reports that training on all speakers gave better validation likelihood than training on one, which suggests the speakers share most of what the network learns.

?

Why is training fast but generation slow?
In training every past sample is known, so the causal convolutions compute all predictions at once and the GPU stays busy. In generation the next input does not exist until the previous step has drawn it, so one second of audio takes 16,000 sequential passes. Caching each layer's past outputs cuts the work per step to one update per layer, and Parallel WaveNet later removed the sequential loop by distilling the model into a feed-forward network.

?

What does a 300 ms receptive field mean for what you hear?
The network can only use the last 300 ms of its own output, about two or three phonemes. Without text conditioning, the VCTK model produced fluent babble: realistic sounds and intonation that never form words or sentences. For text-to-speech, the text conditioning supplies the long-range structure, and the external F0 model supplies the pitch contour that 240 ms of context could not.

?

Can WaveNet recognize speech as well as generate it?
The paper adapted it for TIMIT phoneme recognition by mean-pooling the dilated stack's activations into 10 ms frames, adding a few non-causal convolutions, and training with two losses (next-sample prediction and frame classification). It reached an 18.8% phone error rate on the test set, which the authors describe as the best result at the time from a model trained directly on raw audio.

Footnotes & further reading

  1. The paper: van den Oord et al., WaveNet: A Generative Model for Raw Audio (arXiv, September 2016; presented at the 9th ISCA Speech Synthesis Workshop). Appendix A gives background on statistical parametric synthesis; Appendix B the listening-test details. DeepMind's blog post has the audio samples.
  2. The image models WaveNet adapts: van den Oord, Kalchbrenner & Kavukcuoglu, Pixel Recurrent Neural Networks (ICML 2016), and van den Oord et al., Conditional Image Generation with PixelCNN Decoders (NeurIPS 2016), the source of the gated unit.
  3. Dilated convolutions for dense prediction: Yu & Koltun, Multi-Scale Context Aggregation by Dilated Convolutions (ICLR 2016). Residual connections: He et al., covered in the ResNet explainer.
  4. Cached generation: Paine et al., Fast Wavenet Generation Algorithm (2016). Distillation into a parallel model: van den Oord et al., Parallel WaveNet: Fast High-Fidelity Speech Synthesis (ICML 2018), and DeepMind's announcement (November 2017).
  5. WaveNet as a vocoder for mel spectrograms: Shen et al., Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions (Tacotron 2, ICASSP 2018).
  6. The reimplementation whose channel sizes this page uses: ibab/tensorflow-wavenet (TensorFlow, 2016). DeepMind did not release code for the paper.
  7. Mu-law companding as standardized for telephony: ITU-T Recommendation G.711 (1988).