Learn
Lesson 04 / 13Core20 min

The Encoder: BERT-Base and BERT-Large

A Transformer encoder with no causal mask

By the end of this lesson you can

  • Count BERT's parameters from L, H and A
  • Identify the attention mask as the key difference from GPT
  • List the small changes from the 2017 Transformer (GELU, learned positions, pooler)
Visual explainer
Loading lab
Explanation

Architecturally, BERT is almost boring: it is the Transformer encoder from “Attention Is All You Need”, stacked deeper. What is new is how it is trained, not what it is.

Two Sizes

The paper uses L for layers, H for hidden size and A for attention heads, with a feed-forward size of 4H:

BERT-Base: L = 12, H = 768, A = 12 → 110M parameters
BERT-Large: L = 24, H = 1024, A = 16 → 340M parameters

BERT-Base was deliberately given the same size as OpenAI GPT, so that the comparison isolates the effect of bidirectional training. Both sizes use d_k = H / A = 64 per head.

Where the Parameters Live

Each layer holds about 12H² weights: 4H² for the Q, K, V and output projections and 8H² for the two feed-forward matrices. For BERT-Base that is about 7.1M per layer, or 85M for the stack. The WordPiece embedding table adds 23M, a fifth of the model.

The One Difference from GPT That Matters

GPT is a Transformer decoder stack: its self-attention uses a causal mask so token i attends only to positions ≤ i. BERT drops the causal mask. Every token attends to every other token (except padding) in every layer. Same weights, same compute, a different mask, and a completely different kind of representation.

Small Changes from the 2017 Transformer

BERT uses the GELU activation instead of ReLU in the feed-forward network, learned position embeddings instead of sinusoids, and adds segment embeddings and a pooler (a dense layer with tanh over the [CLS] vector). The post-LN residual layout is unchanged: LayerNorm(x + Sublayer(x)).

Common mistake: calling BERT “an encoder-decoder model”. It has no decoder and no cross-attention. It maps n input tokens to n contextual vectors, which is exactly why it cannot generate text directly.
Interactive code

Edit the code and press Ctrl + Enter to run it in your browser.

pynumpy · matplotlib
Exercises
1

BERT-Large has 24 layers. Would you get more parameters from doubling L or from increasing H from 768 to 1086 (about √2 × 768)?

Key takeaway

BERT-Base (12/768/12, 110M) and BERT-Large (24/1024/16, 340M) are plain Transformer encoders. Removing the causal mask is what makes them bidirectional.

Up next

With full attention, predicting the next word is broken. The fix: masked language modelling.