Learn
Lesson 02 / 13Warm-up20 min

Patches as Tokens

Reshape H×W×C into N = HW/P² flattened patches

By the end of this lesson you can

  • Reshape an image into a sequence of flattened patches x_p ∈ ℝ^(N×(P²·C))
  • Compute N for different resolutions and patch sizes, e.g. 224/16 gives 196 tokens
  • Map between token indices and grid positions in row-major order
Visual explainer
Loading lab
Explanation

A Transformer expects a 1D sequence of tokens. An image is a 3D array of pixels. The title of the paper gives the bridge away: cut the image into 16×16 patches and treat each patch as a word.

From an Image to a Sequence

Section 3.1: the standard Transformer receives as input a 1D sequence of token embeddings. To handle 2D images, the image is reshaped into a sequence of flattened 2D patches:

x ∈ ℝ^(H×W×C)   →   x_p ∈ ℝ^(N×(P²·C)),    N = HW / P²

Here (H, W) is the resolution of the original image, C is the number of channels, (P, P) is the resolution of each patch, and N is the number of patches. N also serves as the effective input sequence length for the Transformer. Each row of x_p is one patch, unrolled into a vector of P·P·C numbers.

The Standard Case: 224 and 16

image 224 x 224 x 3 = 150,528 numbers patch size 16 x 16 grid 224/16 = 14 patches per side N 14 x 14 = 196 patches each patch 16 * 16 * 3 = 768 numbers x_p 196 x 768 = 150,528 numbers (nothing lost)

Patching is just a rearrangement: every pixel ends up in exactly one patch and nothing is thrown away. Note also the coincidence that a 16×16×3 patch has 768 values, the same as the hidden size of ViT-Base. That is a coincidence of the B/16 configuration, not a requirement: the next lesson projects patches to D dimensions anyway.

Row-Major Ordering

The patches are read left to right, top to bottom, like words on a page. Patch index i corresponds to grid position (row, col) = (i div 14, i mod 14). Patch 0 is the top-left corner, patch 13 the top-right, patch 14 the first patch of the second row, and patch 195 the bottom-right. Inside each patch, the pixels are flattened in the same row-major way, with the channels innermost.

The order is a convention, and the Transformer itself does not care about it: as lesson 4 shows, self-attention is blind to the order of its inputs. The only thing that ties a token to its location is the position embedding added at the matching index.

Patch Size Controls Sequence Length

Because N = HW/P², the sequence length is inversely proportional to the square of the patch size. Halving the patch quadruples the number of tokens:

patch size grid at 224 N (tokens) values per patch 32 7 x 7 49 3,072 16 14 x 14 196 768 14 16 x 16 256 588 8 28 x 28 784 192

Smaller patches give the model a finer view of the image, but every extra token makes self-attention more expensive (lesson 6) and every model with smaller patches costs more compute (lesson 7). The paper names models by their patch size for this reason: ViT-B/16 and ViT-B/32 have the same parameters but very different costs.

The same arithmetic explains the resolution change at fine-tuning. The paper fine-tunes at higher resolution than it pre-trains, and keeps the patch size the same. A 384×384 image with 16×16 patches gives a 24×24 grid, so 576 tokens instead of 196. The Transformer can process the longer sequence as is; only the position embeddings need adjusting (lesson 4).

Why Not Pixels?

Treating each pixel as a token would make N = 50,176 for a 224×224 image, and attention over that many tokens would need about 2.5 billion pairwise scores per head per layer. Patches cut that by a factor of 256² = 65,536 for P = 16. They also give each token a meaningful chunk of content: a 16×16 patch can contain an edge, a texture, part of an eye, much as a word carries more meaning than a letter.

Common mistake: reshaping an (H, W, C) image straight to (N, P²·C) with a single reshape. That mixes pixels from different patches, because consecutive memory runs along whole image rows, not along patches. You must first split rows and columns into (grid, within-patch) axes, move the two grid axes to the front, and only then flatten. The code cell checks its fast version against a slow loop for exactly this reason.
Interactive code

Edit the code and press Ctrl + Enter to run it in your browser.

pynumpy · matplotlib
Exercises
1

Images need not be square. Patchify a random 160 × 256 × 3 image with P = 32, report N and the grid shape, and check the round trip with unpatchify.

2

Which token, and which position inside that token's 768-vector, holds pixel (row 100, column 37, channel 2)? Work it out by hand, then verify against xp.

Key takeaway

ViT cuts an image into non-overlapping P×P patches, flattens each, and reads them in row-major order. The sequence length N = HW/P² is inversely proportional to the square of the patch size: 196 tokens for a 224×224 image with 16×16 patches.

Up next

Next: a linear projection turns each raw patch into a D-dimensional token.