Learn
Lesson 05 / 13Core25 min

Masked Language Modelling

Choose 15% of tokens and predict them from both sides

By the end of this lesson you can

  • Describe the MLM corruption procedure precisely
  • Implement 15% selection and the 80/10/10 replacement
  • Explain the cost of predicting only 15% of positions
Visual explainer
Loading lab
Explanation

The masked language model (MLM) is BERT's central idea. It turns the impossible “bidirectional language model” into an easy-to-state training task.

The Procedure

For each training sequence, choose 15% of the WordPiece positions uniformly at random. Never choose [CLS] or [SEP]. At each chosen position, the model must predict the original token from its final hidden vector.

A chosen token is not always replaced by [MASK]. With the paper's example sentence “my dog is hairy”:

80%: my dog is [MASK]
10%: my dog is apple (a random token)
10%: my dog is hairy (unchanged)

In all three cases the target is “hairy”. The next lesson explains why the mix is not 100% [MASK].

What the Model Has to Learn

Because the model never knows which input tokens are corrupted, or even which positions will be tested, it must keep a good contextual representation of every token. A random replacement might be sitting anywhere, so it cannot trust any input blindly.

The Cost: Fewer Predictions per Batch

A left-to-right LM gets a training signal at every position. MLM gets one at only 15% of positions, about 77 per 512-token sequence. The paper shows MLM converges a little more slowly than a left-to-right model, but beats it in accuracy almost from the start of training. The trade is worth it.

Why 15%? Mask too little and training is slow, since each sequence yields few predictions. Mask too much and there is not enough context left to predict from. 15% is the paper's balance point, and it has stayed the default for most successors.
Interactive code

Edit the code and press Ctrl + Enter to run it in your browser.

pynumpy · matplotlib
Exercises
1

Change mask_tokens so it never selects [PAD] positions inside a padded batch and verify it with a batch where the last 30 positions are 0.

Key takeaway

Pick 15% of WordPieces (never [CLS]/[SEP]); replace 80% with [MASK], 10% with a random token, keep 10%; predict the originals at all picked positions.

Up next

Why not always use [MASK]? The answer is the pre-train / fine-tune mismatch.