BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Hide words in a sentence and train a Transformer to fill them in from both sides.

The pre-trained network is then fine-tuned for each task with one small output layer. In October 2018 this set a new state of the art on eleven language benchmarks.

A language model trained left to right gives each word a representation built only from the words before it. BERT gets context from both sides, and the paper is mostly about the one obstacle in the way: a deep network that sees both sides can copy the word it is asked to predict.

By 2018, natural language processing had adopted the recipe computer vision used with ImageNet: pre-train a large network on a large amount of data, then adapt it to the task you care about with a small labeled set. For text the pre-training data needs no labels at all, because a language model can be trained to predict words from other words, and every sentence ever written supplies the answers.

Two ways of using such a model were in circulation. ELMo was feature-based: a pre-trained LSTM (a recurrent network) produced a vector for each word in context, and you fed those vectors as extra inputs into whatever task-specific architecture you already had. OpenAI GPT was a fine-tuning method: a Transformer with almost no task-specific parts, adapted by continuing to train all of its weights on the task. BERT is a fine-tuning method, like GPT. Both GPT and ELMo pre-trained with ordinary language models that read in one direction at a time, and BERT removes that constraint.

BERT is an encoder. It reads a token sequence and returns one vector per position, and a small task layer reads those vectors. It has no decoder, it was never trained to produce the next word, and it cannot be prompted to continue a sentence.

The paper rests on a short list of ideas, taken in this order below:

1. pre-train one network, then fine-tune a copy of it per task with a thin head on top;
2. a deep network that reads both directions can see the word it predicts, through its neighbors;
3. deleting the word from the input (masked language modeling) removes that leak;
4. the [MASK] symbol is not always used, to limit the mismatch with fine-tuning;
5. inputs hold a pair of segments, and a second task (next sentence prediction) trains on pairs;
6. an ablation that holds size and data fixed and isolates the bidirectional objective.

Pre-train once, fine-tune per task

BERT has two phases. In pre-training, the network reads unlabeled text (English Wikipedia, 2,500M words, and the BooksCorpus, 800M words: 3.3 billion words in total) and learns from two tasks that generate their own labels: predict hidden words, and decide whether one sentence follows another. In fine-tuning, you copy the pre-trained weights, attach one output layer for your task, and train everything on your labeled examples for two to four epochs. Each task gets its own fine-tuned copy, and every copy starts from the same pre-trained weights.

Between tasks, only the input format and the output layer change. Sentiment classification, entailment, named-entity tagging and question answering all use the same stack of Transformer layers. Classification reads the output vector at a special first position called [CLS]; tagging and question answering read the output vector at every token. Switch the task in Figure 1 and watch which output each head reads while the encoder stays the same:

Pre-training BERT_LARGE took four days on 16 Cloud TPUs. By contrast, the paper reports that every fine-tuning result in it can be reproduced in at most one hour on a single Cloud TPU, or a few hours on a GPU, starting from the released pre-trained model; the SQuAD model reaches 91.0 dev F1 after about 30 minutes. Every task starts from the same pre-trained vectors, and the next section covers the design decision behind them.

Why a deep model cannot read both ways directly

A standard language model predicts each word from the words before it. GPT works this way, and so does each half of ELMo (one half reads forward, the other backward). The representation such a model builds at a position contains only one side of the sentence. That suits generation, where only the past exists. It hurts understanding, because the words after a word often decide what it means: in "she sat on the bank of the river", the word that separates a riverbank from a lender comes three words later.

The obvious fix is to let every layer attend in both directions and keep predicting each word from the network's output at its position. The paper gives one sentence on why this fails: bidirectional conditioning would allow each word to "indirectly see itself", and the model could trivially predict the target word "in a multi-layered context". The failure comes from stacking layers, and it needs at least two of them.

Walk it on "the cat sat on the mat", predicting sat at position 3. Suppose you even forbid each position from attending to its own input. In layer 1, position 2 (cat) attends to every input, including sat, so its layer-1 vector now carries information about sat. In layer 2, position 3 attends to position 2's layer-1 vector and receives that information. The output at position 3 now depends on the word it is supposed to predict, and training drives the network to copy it rather than to model language. With a single layer there is no such path: position 3's output depends only on the other inputs. Stopping the leak with attention masks would take a different mask for every predicted position, which means a separate forward pass per prediction. Follow the amber path in Figure 2:

This is why earlier pre-training was one-directional: a left-to-right model never has the next word anywhere in its input, so the leak cannot happen. ELMo got both directions by training a left-to-right LSTM and a separate right-to-left LSTM and concatenating their outputs. The two halves never see each other during training, so no single representation weighs both sides jointly. The paper calls this shallow bidirectionality; the word refers to the late concatenation, since each of ELMo's LSTMs is itself several layers deep.

The third mode in the figure is BERT's. If the target word is replaced in the input by a placeholder, no path through the network carries it, however many layers there are, and every other position can attend in both directions at every layer. One forward pass then scores all the replaced positions of a sequence at once.

Masked language modeling

BERT's first pre-training task is the masked language model (MLM). The paper notes it is the Cloze task from a 1953 readability study: delete some words from a passage and ask a reader to fill them in from context. BERT chooses 15% of the token positions in each sequence at random, hides them (the next section covers exactly how), and trains the network to predict the original tokens at those positions.

The tokens are WordPiece units: a fixed vocabulary of subword pieces (30,000 in the paper; the released uncased English vocabulary file has 30,522 entries). Common words are one piece each. A rarer word splits into pieces that are in the vocabulary, with ## marking a piece that continues the previous one: "embeddings" becomes em ##bed ##ding ##s. Any string can be spelled from the pieces, so there are no out-of-vocabulary words, and predicting a hidden token is a classification over a fixed list. Masking applies to pieces, so a hidden piece may be half a word.

At each chosen position $i$, the final hidden vector $\mathbf{T}_i$ goes through a softmax over the vocabulary (the softmax turns one score per vocabulary entry into probabilities that sum to 1), and the loss is the cross-entropy against the original token: minus the log of the probability the model gave that token. With $\mathcal{M}$ the set of chosen positions and $\tilde{\mathbf{x}}$ the corrupted sequence the network reads, the loss is averaged over the chosen positions (the paper's appendix and the released code both use the mean):

The sum runs over chosen positions only. The other 85% of positions are inputs and receive no loss of their own. This separates BERT from a denoising autoencoder, which corrupts its input and then reconstructs all of it; the paper makes the contrast itself. Step the hidden word along the sentence in Figure 3; the arcs show that every prediction draws on context from both sides:

In the released code the prediction head is slightly more than a softmax. Each chosen $\mathbf{T}_i$ passes through a dense layer of width $H$, a GELU and a layer normalization, and is then multiplied by the transpose of the input token-embedding table plus a per-token bias. Reusing the embedding table ties input and output: the score for token $v$ is a dot product with the same vector that represents $v$ on the way in.

With real shapes, one pre-training step at sequence length 128 on BERT_BASE looks like this. The batch holds 256 sequences, so the input ids are a [256, 128] integer tensor, and after the embedding and 12 layers the output is [256, 128, 768]. A full-length sequence has $\text{round}(0.15 \times 128) = 19$ chosen positions, so the head gathers up to 256 × 19 = 4,864 vectors into a [4864, 768] matrix and produces logits of shape [4864, 30522]. Equation (1) averages those 4,864 cross-entropies. The gradient flows back through the head into every layer and into the embeddings of all 32,768 input positions, including the unchosen ones, because every chosen prediction attends to them.

Only 15% of positions produce a loss, where a left-to-right model gets a prediction at every position. The paper checked whether this slows training (Appendix C.1). The masked model converges slightly more slowly in steps, and its downstream accuracy is higher than the left-to-right model's almost from the start of training.

The 80/10/10 split

The [MASK] symbol appears only in pre-training. Fine-tuning feeds complete, uncorrupted sentences. A network pre-trained with [MASK] at every prediction site can come to rely on the symbol, for example by treating it as the marker for "compute a prediction here", and that cue never appears again. The paper names this pre-training/fine-tuning mismatch as the reason for its next choice.

When a position is chosen, BERT replaces it with [MASK] 80% of the time, with a random vocabulary token 10% of the time, and leaves it unchanged 10% of the time. In all three cases the target is the original token. Sample chosen positions in Figure 4 and watch the counts approach the split:

The rates apply to the chosen positions, not to the whole sequence. Out of every 100 tokens, 15 are chosen; 12 of those become [MASK], 1.5 become a random token, and 1.5 are unchanged. The network sees 12% of its input masked and is graded on 15%.

On "the cat sat on the mat" with sat chosen, the input is "the cat [MASK] on the mat" with probability 0.8, something like "the cat apple on the mat" with probability 0.1, and the unchanged sentence with probability 0.1. The label is sat each time, and the loss at that position is the cross-entropy between the model's softmax and sat. The official data generator (create_pretraining_data.py) picks an exact count per sequence, $\text{round}(0.15\,n)$ positions capped at 20 for 128-token data, and never picks [CLS] or [SEP]:

# one masked-LM training example, as create_pretraining_data.py builds it
cand = [i for i, t in enumerate(tokens) if t not in ("[CLS]", "[SEP]")]
shuffle(cand)
n_pred = min(20, max(1, round(0.15 * len(tokens))))  # 20 = max_predictions_per_seq
inputs, labels = list(tokens), {}
for i in cand[:n_pred]:
    labels[i] = tokens[i]            # the target is always the original token
    r = random()
    if r < 0.8:
        inputs[i] = "[MASK]"         # 80% of chosen positions
    elif r < 0.9:
        inputs[i] = tokens[i]        # 10%: left unchanged
    else:
        inputs[i] = random_token()   # 10%: any vocabulary entry
# loss: cross-entropy at the positions in labels only, averaged over them

The paper gives only the mismatch as its reason. A second reading is common in later write-ups but is not in the paper: because a chosen token can look exactly like an ordinary one (the unchanged 10%) or like a wrong word (the random 10%), the network cannot tell from the input which positions it will be graded on, so it has to keep an accurate contextual representation of every token, which is what fine-tuning uses.

Appendix C.2 tests the split directly. It pre-trains BERT_BASE with six different mixes and measures MNLI after fine-tuning and named-entity recognition (NER) both fine-tuned and feature-based, where BERT stays frozen and a separate model reads its last four layers. Figure 5 plots each mix as its change from 80/10/10. Pick all-[MASK] (100/0/0) and compare the three rows:

With fine-tuning, every mix lands within 0.6 points of 80/10/10 on both tasks; the paper calls fine-tuning "surprisingly robust" to the choice. The feature-based column spreads further. All-[MASK] loses 0.9 F1 there (94.9 to 94.0), the largest drop in the table. The paper expected this: a frozen network has no chance to adjust its representations to inputs without [MASK]. All-random (0/0/100) is worse than 80/10/10 on every column, most on MNLI (84.2 to 83.6).

Sentence pairs and next sentence prediction

Many tasks are about two pieces of text: does this hypothesis follow from that premise, does this passage answer that question. A network pre-trained only on single sentences has never had to relate two of them. BERT's input format holds a pair, and its second pre-training task trains on pairs.

Every sequence starts with [CLS]. Two segments, called sentences A and B (each can be any span of contiguous text), are packed with a [SEP] after each. A learned segment embedding marks every token as belonging to A or B, so an attention head can tell which segment a token is in without locating the [SEP]. A learned position embedding marks where each token sits: a table of 512 trained vectors, one per position, where the original Transformer used fixed sine waves. The vector that enters the first layer at position $i$ is the sum of three rows looked up in three tables:

For "[CLS] my dog [SEP] he runs [SEP]", position 4 (he) gets the token row for he, the segment-B row and position row 4, each a vector of length 768, added elementwise.

The second task is next sentence prediction (NSP). For half of the pre-training pairs, B is the text that actually followed A in the source document (label IsNext). For the other half, B is a random sentence from elsewhere in the corpus (NotNext). The final hidden vector at [CLS], written $\mathbf{C}$, feeds a two-way classifier. Toggle sentence B in Figure 6 and watch the input rows and the verdict:

Pre-training minimizes the sum of the two losses, each averaged over its own predictions, where $y$ is the IsNext/NotNext label:

The paper insists on a document-level corpus (Wikipedia text passages, whole books) rather than shuffled single sentences, so that long contiguous sequences can be extracted, and the NSP pairs are cut from those same documents. The final model gets 97 to 98% of NSP pairs right. The paper also warns that $\mathbf{C}$ is not a meaningful sentence representation until it is fine-tuned, since pre-training shaped it only to answer the NSP question.

In BERT's own ablation, removing NSP hurt QNLI (does this sentence answer this question), MNLI (does this premise entail this hypothesis) and SQuAD, most on QNLI (88.4 to 84.9). Later work reversed this: RoBERTa (2019) dropped NSP, packed full documents into each sequence, trained longer, and matched or slightly improved on the downstream results. That is a later finding, not a claim of the BERT paper, and both results hold for their own training setups.

The encoder and its two sizes

The network itself is the encoder from Attention Is All You Need, almost unchanged. What separates it from GPT is which entries of the attention matrix may be nonzero. Attention over $n$ tokens is an $n \times n$ grid: row $i$ is the token doing the looking, column $j$ a token it may look at. GPT is a decoder and zeroes every entry above the diagonal, so each token sees itself and its past. BERT leaves the grid full. Move the query row in Figure 7 and flip the mask:

The paper reports two sizes. BERT_BASE has $L{=}12$ layers, hidden size $H{=}768$ and $A{=}12$ attention heads, 110M parameters in total. It was sized to match OpenAI GPT so that the comparison between them isolates the attention mask. BERT_LARGE has $L{=}24$, $H{=}1024$, $A{=}16$, 340M parameters. Both use a feed-forward inner width of $4H$ (3,072 and 4,096) and the GELU activation, a smooth variant of ReLU, instead of ReLU itself.

The 110M can be reproduced by hand. Each layer has four $768 \times 768$ attention projections (query, key, value, output) and two feed-forward matrices of $768 \times 3072$, about 7.09M weights with biases and layer norms; twelve layers make 85.1M. The token-embedding table is 30,522 × 768 = 23.4M, and positions, segments and a final 768 × 768 pooling layer add about 1M. The total is 109.5M, and the same count for the large model gives 335.1M. About a fifth of BERT_BASE is the vocabulary table.

Pre-training ran 1,000,000 steps at a batch of 256 sequences of up to 512 tokens (the paper's figure is 128,000 tokens per batch, about 40 passes over the 3.3-billion-word corpus). To save compute, 90% of the steps used sequences of length 128 and the last 10% used 512, mainly so the position rows beyond 128 would be trained. The optimizer was Adam with learning rate 1e-4, $\beta_1 = 0.9$, $\beta_2 = 0.999$, weight decay 0.01, a 10,000-step warmup and linear decay, with dropout 0.1 on all layers. The released optimizer applies the weight decay outside Adam's adaptive scaling, the scheme later named AdamW. BERT_BASE trained on 4 Cloud TPUs (16 chips) and BERT_LARGE on 16 Cloud TPUs (64 chips), four days each.

One output layer per task

Fine-tuning maps each task onto the pre-training format. A sentence pair (paraphrase, entailment, question and passage) fills segments A and B. A single sentence fills A and leaves B empty. The output side has two patterns: sequence-level tasks read $\mathbf{C}$, and token-level tasks read every $\mathbf{T}_i$.

For classification with $K$ labels the only new parameters are a matrix $W \in \mathbb{R}^{K \times H}$, and the loss is the usual log-softmax:

For SST-2 sentiment on BERT_BASE, $K = 2$ and $W$ holds 1,536 numbers, against 110M pre-trained ones. Every weight gets gradient, not only $W$:

# sentence classification (SST-2: K = 2 labels)
T = bert(tokens, segments)           # [n, H]: one output vector per position
C = T[0]                             # [H]: the output at [CLS]
logits = W @ C                       # W: [K, H], the only new weights
loss = -log_softmax(logits)[label]
loss.backward()                      # gradient reaches W AND every BERT weight

In the released code, the classifier and the NSP head read a pooled vector, a dense layer with tanh applied to $\mathbf{C}$ whose weights come from pre-training. The paper describes $\mathbf{C}$ itself as the input. For named-entity tagging, a classifier over the tag set reads each token's $\mathbf{T}_i$, using the first piece of each word.

Question answering needs only two new vectors. In SQuAD v1.1 the input is a question (segment A) and a Wikipedia passage (segment B), and the model must point at the span of the passage that answers it. Fine-tuning adds a start vector $\mathbf{S} \in \mathbb{R}^{H}$ and an end vector $\mathbf{E} \in \mathbb{R}^{H}$. Each passage token $i$ gets a start probability, a softmax of dot products over the passage:

The end probability uses the same formula with $\mathbf{E}$. Training maximizes the log-likelihood of the true start plus that of the true end. At prediction time a span from $i$ to $j$ scores $\mathbf{S}\cdot\mathbf{T}_i + \mathbf{E}\cdot\mathbf{T}_j$, and the prediction is the highest-scoring span that ends at or after its start:

The two softmaxes are independent, so without the $j \ge i$ condition the best start could lie after the best end. SQuAD v2.0 adds questions the passage does not answer. BERT handles them by treating "no answer" as a span that starts and ends at [CLS], with score $s_{\text{null}} = \mathbf{S}\cdot\mathbf{C} + \mathbf{E}\cdot\mathbf{C}$, and it gives a real answer only when $\hat{s} > s_{\text{null}} + \tau$, with the threshold $\tau$ tuned on the dev set. In Figure 8, set the start and end yourself, try an end before the start, then switch to the question the passage cannot answer:

For the answerable question, march has the largest start logit (4.0) and 1889 the largest end logit (4.2), so the predicted span is "march 1889" with score 8.2 against a no-answer score of 1.8. For the unanswerable one, the best span scores 2.2 and [CLS] scores 6.8, so the prediction is "no answer". The whole procedure in code:

# SQuAD: two new vectors S and E, each of length H
T = bert(question_and_passage)       # [n, H]
start = T @ S                        # [n]: start logit S.T_i at every position
end = T @ E                          # [n]: end logit E.T_j
best, i, j = max((start[a] + end[b], a, b)
                 for a in passage for b in passage if b >= a)
null = start[0] + end[0]             # position 0 is [CLS]; SQuAD 2.0 only
answer = tokens[i : j + 1] if best > null + tau else "no answer"
# training: maximize log softmax(start)[s_true] + log softmax(end)[e_true]

The paper found batch size 16 or 32, learning rate 5e-5, 3e-5 or 2e-5, and 2 to 4 epochs to work across tasks, and suggests a grid search over those values. For GLUE it used batch 32 for 3 epochs and picked the learning rate on the dev set; BERT_LARGE was sometimes unstable on small datasets, so it ran several random restarts and kept the best on dev.

Results and the ablation

BERT set a new state of the art on eleven tasks. The headline numbers:

• GLUE, a suite of nine sentence-level tasks: BERT_LARGE scored 80.5 on the official leaderboard against OpenAI GPT's 72.8, a gain of 7.7 points. The paper's own GLUE table averages without the WNLI task and gives 82.1 against 75.1; the two averages should not be mixed. On MNLI, the largest GLUE task, the gain is 4.6 points (86.7 against 82.1).
• SQuAD v1.1 ("dev" below is the public development split used for tuning, "test" the hidden leaderboard split): the best system, an ensemble also fine-tuned on the TriviaQA dataset first, reached 93.2 test F1, 1.5 above the top leaderboard entry. A single BERT_LARGE reaches 90.9 dev F1. (F1 here measures word overlap between the predicted and true answer spans.)
• SQuAD v2.0: 83.1 test F1, 5.1 points above the previous best system.
• SWAG, choosing the most plausible continuation of a sentence: 86.3, against 78.0 for OpenAI GPT. A single human expert scores 85.0 and five annotators together 88.0.

Those numbers mix several changes at once: a bidirectional objective, NSP, a larger model, and more data than GPT used. Section 5.1 separates them. It keeps the BERT_BASE architecture, pre-training data, fine-tuning scheme and hyperparameters fixed and changes only the pre-training task: full BERT, then no NSP, then a left-to-right language model (LTR) without NSP, and finally that LTR model with a randomly initialized BiLSTM (a bidirectional LSTM layer) added on top during fine-tuning. Switch tasks in Figure 9 and compare the third bar with the first:

On SST-2 sentiment, going left-to-right costs 0.6 points (92.7 to 92.1): the polarity of a review is usually clear from a left-to-right read. On SQuAD it costs 10.7 F1 (88.5 to 77.8), and on MRPC paraphrase detection 9.2 points (86.7 to 77.5). A span answer has to be located using words on both sides of it, and each token of the left-to-right model has none of its right context. The added BiLSTM recovers part of the SQuAD loss (84.9) but stays 3.6 below full BERT, and it hurts on the GLUE tasks. The losses are largest on the tasks where right context should matter most, which is the paper's evidence that the bidirectional objective, rather than size or data, produced the gains.

Model size and frozen features

Section 5.2 pre-trains six encoder shapes with the same procedure and fine-tunes each on three GLUE tasks (dev accuracy averaged over five fine-tuning restarts). The paper's point concerns MRPC, which has only 3,600 labeled training examples. Drag across the sizes in Figure 10 and follow the MRPC line:

Held-out perplexity falls from 5.84 to 3.23. Perplexity is the exponential of the masked-LM cross-entropy, so at 3.23 the model is, on average, as uncertain as a uniform choice among 3.23 tokens. Every task improves at every step, including MRPC, from 79.8 at 3 layers to 87.8 at BERT_LARGE. The second and third configurations have the same parameter count (the number of heads does not change it) and differ only in splitting attention into 3 or 12 heads; 12 heads gains 1.3 points on MNLI-m. The paper describes itself as the first work to show convincingly that scaling to extreme sizes helps on very small tasks. Its hypothesis for why is fine-tuning: the task adds very few randomly initialized parameters, so a small labeled set only has to adjust a large representation. Earlier size studies used the feature-based approach and got mixed results.

Section 5.3 tests the feature-based approach with BERT anyway, on CoNLL-2003 named-entity recognition. BERT_BASE stays frozen; activations from one or more layers feed a randomly initialized two-layer BiLSTM of width 768 and a classifier. Frozen features suit tasks that need their own architecture, and they let you compute the expensive representation once and run many cheap experiments on it. Figure 11 compares the choices of layer:

The embeddings alone, before any contextual layer, give 91.0 F1. Concatenating the top four layers gives 96.1, 0.3 F1 behind fine-tuning the whole model (96.4). The last layer alone (94.9) scores below the second-to-last (95.6). BERT therefore works as a frozen feature extractor in the ELMo style as well as a fine-tuned model, which is also the setting where the masking mix in Figure 5 mattered most.