Masked Language Modelling
Choose 15% of tokens and predict them from both sides
By the end of this lesson you can
- Describe the MLM corruption procedure precisely
- Implement 15% selection and the 80/10/10 replacement
- Explain the cost of predicting only 15% of positions
The masked language model (MLM) is BERT's central idea. It turns the impossible “bidirectional language model” into an easy-to-state training task.
The Procedure
For each training sequence, choose 15% of the WordPiece positions uniformly at random. Never choose [CLS] or [SEP]. At each chosen position, the model must predict the original token from its final hidden vector.
A chosen token is not always replaced by [MASK]. With the paper's example sentence “my dog is hairy”:
10%: my dog is apple (a random token)
10%: my dog is hairy (unchanged)
In all three cases the target is “hairy”. The next lesson explains why the mix is not 100% [MASK].
What the Model Has to Learn
Because the model never knows which input tokens are corrupted, or even which positions will be tested, it must keep a good contextual representation of every token. A random replacement might be sitting anywhere, so it cannot trust any input blindly.
The Cost: Fewer Predictions per Batch
A left-to-right LM gets a training signal at every position. MLM gets one at only 15% of positions, about 77 per 512-token sequence. The paper shows MLM converges a little more slowly than a left-to-right model, but beats it in accuracy almost from the start of training. The trade is worth it.
Edit the code and press Ctrl + Enter to run it in your browser.
Change mask_tokens so it never selects [PAD] positions inside a padded batch and verify it with a batch where the last 30 positions are 0.
Key takeaway
Pick 15% of WordPieces (never [CLS]/[SEP]); replace 80% with [MASK], 10% with a random token, keep 10%; predict the originals at all picked positions.
Up next
Why not always use [MASK]? The answer is the pre-train / fine-tune mismatch.