WordPiece Tokenization
Greedy longest-match-first sub-words and the ## continuation marker
By the end of this lesson you can
- Run the basic tokenizer and WordPiece stages by hand
- Implement greedy longest-match-first
- Know when a word becomes [UNK]
Before BERT sees a sentence, the text becomes a sequence of WordPiece tokens. The tokenizer decides what the model can and cannot represent, so it is worth understanding exactly.
Two Stages
First a basic tokenizer lowercases the text (for the uncased model), splits on whitespace and splits off punctuation. Then a WordPiece tokenizer breaks every word into pieces from a fixed vocabulary of 30,000 entries.
Greedy Longest-Match-First
For each word, WordPiece takes the longest prefix that is in the vocabulary, emits it, and repeats on what is left. Pieces that continue a word carry a ## prefix, so the model can tell “able” at the start of a word from “##able” at the end of one.
embeddings → em · ##bed · ##ding · ##s
playing → play · ##ing
If some position has no matching piece at all, the entire word becomes [UNK]. Because every single character is in the vocabulary, that almost never happens for ordinary text.
Why Sub-words?
A word-level vocabulary cannot cover rare words, names and typos, and every unknown word becomes the same useless [UNK]. A character-level model has no unknowns but produces very long sequences, and attention cost grows with the square of the length. Sub-words sit in between: frequent words stay whole, rare words decompose into pieces the model has seen many times.
Consequences Later in the Paper
Masking is applied to WordPieces, not whole words, so “hair” can be masked while “##y” stays visible. For tagging tasks such as NER, a word with several pieces gets one label, attached to its first piece. Both details come back in later lessons.
Edit the code and press Ctrl + Enter to run it in your browser.
Add a vocabulary limit: remove '##ding' and '##bed' and tokenize 'embeddings' again. How many pieces do you get now?
Key takeaway
WordPiece splits each word into the longest vocabulary pieces it can, with ## marking continuations. A 30k vocabulary covers almost any text without unknown words.
Up next
Next: how one or two sentences of tokens are packed into a single input.