Lesson 01 / 13Warm-up20 min

Why BERT: The Unidirectional Bottleneck

Feature-based vs fine-tuning, and why left-to-right language models waste half the context

By the end of this lesson you can

Contrast the feature-based (ELMo) and fine-tuning (GPT) approaches
Explain why a left-to-right LM cannot use right-hand context
See why a naive bidirectional LM lets each word see itself

Visual explainer

Lab · Which context can a model use?

Click a word to make it the blank, then switch between left-to-right, right-to-left, ELMo-style and BERT-style context and watch what each one can see and guess.

Sentence

layers3

what feeds the prediction for “store” · BERT

fill the blank (toy cue model)8/8 words usable

store96%

park1%

bank1%

gym1%

moon1%

Both sides are used jointly in every layer, so right-hand cues like “buy milk” sharpen the guess.

A left-to-right LM can only condition on the words before the blank, so “buy milk” is invisible to it. ELMo trains a forward and a backward LSTM separately and only joins them at the top, so no layer ever mixes both sides. BERT lets every layer attend both ways, which is only safe because the blank is replaced by [MASK]; turn masking off and the word can simply read itself.

Loading lab

Explanation

By 2018 the Transformer had won machine translation. The next question was bigger: could one model, trained once on plain text, become the starting point for every language task? BERT's answer rests on one observation about direction.

Two Ways to Reuse a Pre-trained Model

The paper sorts earlier work into two strategies. The feature-based approach (ELMo) trains a language model, then feeds its internal vectors as extra features into a task-specific architecture. The fine-tuning approach (OpenAI GPT) adds a small output layer to the pre-trained network and trains everything end to end on the task.

Both learned from a standard language model objective: predict the next word from the words before it. That choice is the bottleneck.

The Unidirectional Bottleneck

A left-to-right language model factorises the probability of a sentence as:

P(w₁, \dots, wₙ) = \prodᵢ P(wᵢ | w₁, \dots, wᵢ₋₁)

So when GPT builds the vector for token i, it can only look left. In “the man went to the ____ to buy milk”, the words that settle the blank (buy milk) are on the right, and GPT never sees them. For generation that is the point. For understanding tasks such as question answering, where the whole passage is available, it is a needless handicap.

ELMo tried to fix this by training a forward LSTM and a separate backward LSTM, then concatenating their outputs. But the two directions never interact inside the network: each layer is still one-directional. The paper calls this shallow bidirectionality.

Why Not Just Make the LM Bidirectional?

Take a Transformer encoder with no causal mask and train it to predict each word. After one layer every position has attended to every other, including the word it is supposed to predict. After two layers the word can reach itself through a neighbour. The model would learn to copy, not to understand. This is why nobody trained deep bidirectional language models before BERT.

BERT's Move

Change the task instead of the architecture. Hide some input words behind a [MASK] token and predict only those. Now full self-attention is safe: the answer is simply not in the input. The paper borrows the idea from the Cloze test (Taylor, 1953), the fill-in-the-blank exercise used in reading comprehension.

The key insight to carry forward: left-to-right models throw away half the context. Masking lets a Transformer encoder condition on both sides in every layer without letting a word see itself.

Interactive code

Edit the code and press Ctrl + Enter to run it in your browser.

pynumpy · matplotlib

import numpy as np

tokens = ["the", "man", "went", "to", "the", "[?]", "to", "buy", "milk"]
n = len(tokens)
blank = tokens.index("[?]")

# Which positions may position i read from, in one layer?
ltr  = np.tril(np.ones((n, n), dtype=int))      # GPT: only the past (and itself)
bidi = np.ones((n, n), dtype=int)               # BERT: everything

def reach(mask, layers):
    """Which inputs can influence each output after stacking this many layers."""
    r = np.eye(n, dtype=int)
    for _ in range(layers):
        r = ((mask @ r) > 0).astype(int)
    return r

print("Context visible to the blank at position", blank)
print("  left-to-right:", [t for j, t in enumerate(tokens) if ltr[blank, j] and j != blank])
print("  bidirectional:", [t for j, t in enumerate(tokens) if bidi[blank, j] and j != blank])
print()

# The trap: a naive bidirectional LM lets a word see ITSELF.
# Predicting token i from layer-2 output at position i:
r2 = reach(bidi, 2)
print("After 2 bidirectional layers, can position", blank, "see its own input?", bool(r2[blank, blank]))
print("-> it could just copy the answer. BERT fixes this by replacing it with [MASK].")
print()

# Left-to-right predicts token i from the output at i-1, which never sees i.
r_ltr = reach(ltr, 12)
print("Left-to-right, 12 layers: output at", blank - 1, "sees position", blank, "?", bool(r_ltr[blank - 1, blank]))

import numpy as np

tokens = ["the", "man", "went", "to", "the", "[?]", "to", "buy", "milk"]
n = len(tokens)
blank = tokens.index("[?]")

# Which positions may position i read from, in one layer?
ltr  = np.tril(np.ones((n, n), dtype=int))      # GPT: only the past (and itself)
bidi = np.ones((n, n), dtype=int)               # BERT: everything

def reach(mask, layers):
    """Which inputs can influence each output after stacking this many layers."""
    r = np.eye(n, dtype=int)
    for _ in range(layers):
        r = ((mask @ r) > 0).astype(int)
    return r

print("Context visible to the blank at position", blank)
print("  left-to-right:", [t for j, t in enumerate(tokens) if ltr[blank, j] and j != blank])
print("  bidirectional:", [t for j, t in enumerate(tokens) if bidi[blank, j] and j != blank])
print()

# The trap: a naive bidirectional LM lets a word see ITSELF.
# Predicting token i from layer-2 output at position i:
r2 = reach(bidi, 2)
print("After 2 bidirectional layers, can position", blank, "see its own input?", bool(r2[blank, blank]))
print("-> it could just copy the answer. BERT fixes this by replacing it with [MASK].")
print()

# Left-to-right predicts token i from the output at i-1, which never sees i.
r_ltr = reach(ltr, 12)
print("Left-to-right, 12 layers: output at", blank - 1, "sees position", blank, "?", bool(r_ltr[blank - 1, blank]))

Exercises

Using reach(), find how many bidirectional layers it takes before EVERY position can see its own input through some other position. Does masking the target fix it for any depth?

Key takeaway

Standard LMs are one-directional by construction. BERT gets deep bidirectional context by changing the task (predict masked words) instead of the architecture.

Up next

First, how raw text becomes the tokens BERT actually sees.