The Class Token and Position Embeddings
Eq. 1: z₀ = [x_class; x_p¹E; …; x_pᴺE] + E_pos
By the end of this lesson you can
- Assemble z₀ from the class token, patch embeddings and position embeddings
- Show that self-attention without positions treats an image as a bag of patches
- Read the Appendix D.4 ablation and explain why 1D embeddings suffice
Patch embeddings alone are not enough. The Transformer needs a place to collect a summary of the whole image, and it needs some way to know where each patch came from. ViT borrows both answers from BERT: a learnable class token and learnable position embeddings.
Equation 1
The full input to the encoder is:
The square brackets stack N + 1 row vectors into a matrix. The first row is the class token; the next N rows are the patch embeddings of lesson 3. E_pos has one learnable row for each of the N + 1 slots, added element-wise. For ViT-B/16 at 224×224, z₀ is a 197 × 768 matrix.
The Class Token
Similar to BERT's [class] token, a learnable embedding is prepended to the sequence of embedded patches (z₀⁰ = x_class). It corresponds to no part of the image. Through the layers it attends to all the patches, and its state at the output of the encoder, z_L⁰, serves as the image representation y (Eq. 4, lesson 5). A classification head is attached to it: an MLP with one hidden layer during pre-training, and a single linear layer at fine-tuning time.
The class token is a design choice, not a necessity. Appendix D.3 compares it with global average pooling over the patch outputs, the usual choice in CNNs. Both work similarly well, but they need different learning rates: with a learning rate tuned for the class token, average pooling looks much worse at first. The paper keeps the class token to stay close to the original Transformer, not because it is better.
Why Positions Are Needed At All
Self-attention is permutation equivariant. If you shuffle the input tokens, every output is computed by the same formula from the same set of inputs, so the outputs come out shuffled the same way and are otherwise unchanged. The class token, which attends to all patches with weights that depend only on their content, produces exactly the same output whatever order the patches are in. Without position information, ViT sees an image as a bag of patches: a correctly assembled face and the same 196 patches scrambled at random are indistinguishable. The code cell verifies both statements numerically.
Adding E_pos breaks the symmetry. The row added at index i is tied to the slot, not to the content, so once a patch is moved, it receives a different position vector and the outputs change.
Learnable 1D Embeddings
The paper uses standard learnable 1D position embeddings: one free vector per index 0, 1, …, N, initialised randomly and trained with the rest of the model. They know nothing about rows and columns; index 14 is simply the fifteenth slot. Any 2D structure must be discovered from data. Remarkably, it is. Figure 7 (center) shows the cosine similarity between the learned position embeddings: closer patches tend to have more similar embeddings, and a row-column structure appears, with patches in the same row or column having similar embeddings. The lab above lets you compare that structure with the random embeddings a model starts from.
Appendix D.4: Does the Kind of Embedding Matter?
The authors ablate four ways of encoding position: none at all; 1D embeddings as in the main model; 2D-aware embeddings, where the model learns one set of embeddings for the X coordinate and one for the Y coordinate, each of size D/2, and concatenates them; and relative positional embeddings, which encode the offset between pairs of patches inside attention. Table 8 reports few-shot linear evaluation accuracy:
Having no positional information costs about three points. Having some matters; which kind barely does. The paper's explanation: the encoder works on patches, not pixels, so the spatial grid is small (14×14 instead of 224×224), and learning to represent positions at that resolution is equally easy with any of these strategies. Simplest wins, so ViT uses 1D.
Changing Resolution
Learned embeddings have a fixed number of rows, N + 1. When the paper fine-tunes at higher resolution with the same patch size, the sequence gets longer and the pre-trained embeddings no longer fit. The fix is to arrange the patch embeddings back on their 2D grid and perform a 2D interpolation to the new grid size. This, together with patch extraction, is the only point where the 2D structure of the image is fed into ViT by hand (exercise 1).
Edit the code and press Ctrl + Enter to run it in your browser.
Fine-tuning at 384 × 384 with P = 16 needs 24 × 24 = 576 patch position embeddings, but pre-training learned 14 × 14. Implement the 2D interpolation: reshape E_pos[1:] to (14, 14, D), bilinearly resize to (24, 24, D), and re-attach the class row.
Appendix D.3 compares the class token with global average pooling (GAP) of the patch outputs. Using attn, check whether GAP is also unchanged by scrambling the patches, both without and with position embeddings.
Key takeaway
A learnable class token is prepended and learnable 1D position embeddings are added to all N + 1 tokens. Without them attention is permutation equivariant and the class token cannot tell a scrambled image from the original; with any of 1D, 2D or relative embeddings, results are nearly the same.
Up next
z₀ now goes through the Transformer encoder, the subject of the next lesson.