Patch Embedding
A trainable linear projection E, which is a convolution with kernel = stride = P
By the end of this lesson you can
- Write the patch embedding x_p E with E ∈ ℝ^((P²·C)×D)
- Show that it equals a convolution with kernel size and stride P
- Count the parameters of E for the paper's model variants
A flattened patch is a vector of raw pixel values. Before the Transformer sees it, each patch is mapped to a D-dimensional embedding by one learned matrix. This single linear layer is the only part of ViT that looks at pixels directly.
A Trainable Linear Projection
The Transformer uses a constant latent vector size D through all of its layers. So the paper flattens the patches and maps them to D dimensions with a trainable linear projection E:
The output of this projection is called the patch embeddings. The same E is applied to every patch, so the whole step is a single matrix multiplication of the (N × P²C) patch matrix by the (P²C × D) matrix E, producing an (N × D) sequence. In practice a bias is added too.
It Is a Convolution in Disguise
Look at what E does to one patch: each of the D output values is a weighted sum over all P·P·C pixels of the patch, with weights that do not depend on where the patch is. That is precisely a convolution with D filters of size P×P, applied with a stride of P so the windows tile the image without overlapping:
E[(a, b, c), d] = W_conv[d, c, a, b]
The paper describes this case explicitly when discussing the hybrid model: the patch embedding projection E is applied to patches extracted from a CNN feature map, and in the special case of 1×1 patches it amounts to flattening the feature map and projecting it to D dimensions. Public implementations usually write the patch embedding as a strided convolution because it is fast and needs no explicit reshaping. The code cell builds both and checks they give the same tokens.
Parameter Counts
These are tiny next to the encoder: ViT-B/16 has about 86 million parameters, of which the patch embedding is well under 1%. Notice the trade-off between the two B models. ViT-B/32 has four times the embedding weights of ViT-B/16 (bigger patches mean longer input vectors) but a sequence four times shorter.
What the Filters Learn
Since E is linear, each of its D columns is a filter over a patch and can be shown as a small image. The paper does this in Figure 7 (left) for ViT-L/32: the top principal components of the learned embedding filters resemble plausible basis functions for a low-dimensional representation of the fine structure within each patch. They look like smooth gradients, edges and stripes at various orientations and frequencies, much like the first layer of a CNN.
The code cell shows the idea with three hand-made filters instead of learned ones: the mean colour of a patch, a left-versus-right gradient and a top-versus-bottom gradient. Projecting all 196 patches onto them gives three 14×14 “feature maps”, one value per patch.
Linear, and Only Once
Everything finer than a patch must be captured by this one linear map. There is no nonlinearity between pixels and tokens, and after this step the model never sees pixels again. Spatial detail inside a patch can survive only in the D numbers of its embedding. That is one reason the patch size matters so much for accuracy: smaller patches mean less within-patch detail to compress into each token.
Edit the code and press Ctrl + Enter to run it in your browser.
ViT's patches do not overlap because stride = kernel = P. Run conv2d with kernel 16 and stride 8 instead. How many tokens does the image produce, and how many times is each interior pixel used?
In the hybrid model, patches are taken from a CNN feature map instead of raw pixels, and the paper notes they can be 1×1. Take a random 14 × 14 × 1024 feature map, patchify it with P = 1, and project it to D = 768. What are N and the size of E?
Key takeaway
Each flattened patch is multiplied by the same learned matrix E to give a D-dimensional patch embedding. This is exactly a strided convolution with kernel = stride = P, it holds well under 1% of ViT-Base's parameters, and it is the only place the model sees raw pixels.
Up next
Patch embeddings alone carry no position and no image summary. Next: the class token and E_pos.