The Encoder: BERT-Base and BERT-Large
A Transformer encoder with no causal mask
By the end of this lesson you can
- Count BERT's parameters from L, H and A
- Identify the attention mask as the key difference from GPT
- List the small changes from the 2017 Transformer (GELU, learned positions, pooler)
Architecturally, BERT is almost boring: it is the Transformer encoder from “Attention Is All You Need”, stacked deeper. What is new is how it is trained, not what it is.
Two Sizes
The paper uses L for layers, H for hidden size and A for attention heads, with a feed-forward size of 4H:
BERT-Large: L = 24, H = 1024, A = 16 → 340M parameters
BERT-Base was deliberately given the same size as OpenAI GPT, so that the comparison isolates the effect of bidirectional training. Both sizes use d_k = H / A = 64 per head.
Where the Parameters Live
Each layer holds about 12H² weights: 4H² for the Q, K, V and output projections and 8H² for the two feed-forward matrices. For BERT-Base that is about 7.1M per layer, or 85M for the stack. The WordPiece embedding table adds 23M, a fifth of the model.
The One Difference from GPT That Matters
GPT is a Transformer decoder stack: its self-attention uses a causal mask so token i attends only to positions ≤ i. BERT drops the causal mask. Every token attends to every other token (except padding) in every layer. Same weights, same compute, a different mask, and a completely different kind of representation.
Small Changes from the 2017 Transformer
BERT uses the GELU activation instead of ReLU in the feed-forward network, learned position embeddings instead of sinusoids, and adds segment embeddings and a pooler (a dense layer with tanh over the [CLS] vector). The post-LN residual layout is unchanged: LayerNorm(x + Sublayer(x)).
Edit the code and press Ctrl + Enter to run it in your browser.
BERT-Large has 24 layers. Would you get more parameters from doubling L or from increasing H from 768 to 1086 (about √2 × 768)?
Key takeaway
BERT-Base (12/768/12, 110M) and BERT-Large (24/1024/16, 340M) are plain Transformer encoders. Removing the causal mask is what makes them bidirectional.
Up next
With full attention, predicting the next word is broken. The fix: masked language modelling.