Patches as Tokens
Reshape H×W×C into N = HW/P² flattened patches
By the end of this lesson you can
- Reshape an image into a sequence of flattened patches x_p ∈ ℝ^(N×(P²·C))
- Compute N for different resolutions and patch sizes, e.g. 224/16 gives 196 tokens
- Map between token indices and grid positions in row-major order
A Transformer expects a 1D sequence of tokens. An image is a 3D array of pixels. The title of the paper gives the bridge away: cut the image into 16×16 patches and treat each patch as a word.
From an Image to a Sequence
Section 3.1: the standard Transformer receives as input a 1D sequence of token embeddings. To handle 2D images, the image is reshaped into a sequence of flattened 2D patches:
Here (H, W) is the resolution of the original image, C is the number of channels, (P, P) is the resolution of each patch, and N is the number of patches. N also serves as the effective input sequence length for the Transformer. Each row of x_p is one patch, unrolled into a vector of P·P·C numbers.
The Standard Case: 224 and 16
Patching is just a rearrangement: every pixel ends up in exactly one patch and nothing is thrown away. Note also the coincidence that a 16×16×3 patch has 768 values, the same as the hidden size of ViT-Base. That is a coincidence of the B/16 configuration, not a requirement: the next lesson projects patches to D dimensions anyway.
Row-Major Ordering
The patches are read left to right, top to bottom, like words on a page. Patch index i corresponds to grid position (row, col) = (i div 14, i mod 14). Patch 0 is the top-left corner, patch 13 the top-right, patch 14 the first patch of the second row, and patch 195 the bottom-right. Inside each patch, the pixels are flattened in the same row-major way, with the channels innermost.
The order is a convention, and the Transformer itself does not care about it: as lesson 4 shows, self-attention is blind to the order of its inputs. The only thing that ties a token to its location is the position embedding added at the matching index.
Patch Size Controls Sequence Length
Because N = HW/P², the sequence length is inversely proportional to the square of the patch size. Halving the patch quadruples the number of tokens:
Smaller patches give the model a finer view of the image, but every extra token makes self-attention more expensive (lesson 6) and every model with smaller patches costs more compute (lesson 7). The paper names models by their patch size for this reason: ViT-B/16 and ViT-B/32 have the same parameters but very different costs.
The same arithmetic explains the resolution change at fine-tuning. The paper fine-tunes at higher resolution than it pre-trains, and keeps the patch size the same. A 384×384 image with 16×16 patches gives a 24×24 grid, so 576 tokens instead of 196. The Transformer can process the longer sequence as is; only the position embeddings need adjusting (lesson 4).
Why Not Pixels?
Treating each pixel as a token would make N = 50,176 for a 224×224 image, and attention over that many tokens would need about 2.5 billion pairwise scores per head per layer. Patches cut that by a factor of 256² = 65,536 for P = 16. They also give each token a meaningful chunk of content: a 16×16 patch can contain an edge, a texture, part of an eye, much as a word carries more meaning than a letter.
Edit the code and press Ctrl + Enter to run it in your browser.
Images need not be square. Patchify a random 160 × 256 × 3 image with P = 32, report N and the grid shape, and check the round trip with unpatchify.
Which token, and which position inside that token's 768-vector, holds pixel (row 100, column 37, channel 2)? Work it out by hand, then verify against xp.
Key takeaway
ViT cuts an image into non-overlapping P×P patches, flattens each, and reads them in row-major order. The sequence length N = HW/P² is inversely proportional to the square of the patch size: 196 tokens for a 224×224 image with 16×16 patches.
Up next
Next: a linear projection turns each raw patch into a D-dimensional token.