The Transformer Encoder
Eqs. 2-4: pre-norm residual blocks of MSA and a GELU MLP
By the end of this lesson you can
- Write Eqs. 2 to 4 and say where LayerNorm and the residual connections go
- Implement LayerNorm, GELU and the two-layer MLP
- Explain why a pre-norm encoder defaults to the identity and count 12D² + 13D parameters per layer
Once z₀ is built, the image is just a sequence, and ViT hands it to an almost unmodified Transformer encoder. Three equations describe the entire rest of the model.
Equations 2 to 4
The encoder consists of alternating layers of multiheaded self-attention (MSA, lesson 6) and MLP blocks. In the paper's words: Layernorm (LN) is applied before every block, and residual connections after every block. Each encoder layer ℓ maps an (N+1) × D matrix to another (N+1) × D matrix, so the shape never changes and the layers can be stacked as deep as you like. After the last layer, only row 0 (the class token) is used: it is normalised once more and becomes the image representation y.
Two Sub-blocks, Two Jobs
- MSA mixes information across tokens. It is the only place where one patch can influence another.
- The MLP processes each token on its own. The same two-layer network is applied to every row independently, so it is position-wise: the paper notes that the MLP layers are the local and translationally equivariant part of ViT.
The MLP and GELU
The MLP contains two layers with a GELU non-linearity:
M is the “MLP size” column of Table 1, four times D in every variant (3072 for ViT-Base). GELU, the Gaussian Error Linear Unit, is x·Φ(x) with Φ the standard normal CDF. It behaves like ReLU for large positive or negative inputs but is smooth around zero and lets small negative values through with a small negative output. BERT and GPT use it too; the code cell compares it with ReLU.
Layer Normalisation
LayerNorm normalises each token vector over its own D features: subtract the mean of that row, divide by its standard deviation, then apply a learned per-feature scale and shift. Unlike batch normalisation, nothing depends on other images in the batch or on other tokens in the sequence, so training and inference behave identically.
Pre-norm Residual Blocks
The original Transformer of Vaswani et al. (2017) placed LN after the residual addition (post-norm): z ← LN(z + Sublayer(z)). ViT, following later practice, puts LN inside the branch, before the sublayer (pre-norm). The difference is structural. In pre-norm, the residual stream z is never normalised along its main path: each block reads a normalised copy and adds its result back. If every branch outputs zero, the whole encoder is exactly the identity, the same property that made ResNet blocks easy to optimise. Post-norm cannot do this, because the stream passes through an LN at every layer (exercise 1). Pre-norm Transformers are generally found to be easier to train at depth; the paper simply adopts the design.
Because the stream itself is not normalised, Eq. 4 needs one final LN before the head. That is why y = LN(z_L⁰) rather than z_L⁰ itself.
Parameters per Layer
For ViT-Base (D = 768) that is about 7.09 million parameters per layer, and two thirds of them sit in the MLP. Twelve layers give about 85 million, nearly all of the model's 86 million. Lesson 7 uses this count to reproduce Table 1.
Edit the code and press Ctrl + Enter to run it in your browser.
The original Transformer used post-norm: z ← LN(z + Sublayer(z)). Implement a post-norm block, zero the branch outputs (Wo, bo, W2, b2) in all six blocks, and pass an input with mean 3 and std 2 through both encoders. Which one is the identity?
The cell implements GELU with the tanh approximation. Compare it with the exact x·Φ(x) = 0.5·x·(1 + erf(x/√2)) on [−6, 6]: report the largest difference and the minimum value of GELU and where it occurs.
Key takeaway
Each encoder layer computes z′ = MSA(LN(z)) + z and then z = MLP(LN(z′)) + z′, keeping the (N+1)×D shape. LayerNorm comes before each block and the residual addition after it, and the final representation is y = LN(z_L⁰).
Up next
The one sub-block that lets patches talk to each other is multi-head self-attention, next.