Learn
Lesson 03 / 13Core20 min

Patch Embedding

A trainable linear projection E, which is a convolution with kernel = stride = P

By the end of this lesson you can

  • Write the patch embedding x_p E with E ∈ ℝ^((P²·C)×D)
  • Show that it equals a convolution with kernel size and stride P
  • Count the parameters of E for the paper's model variants
Visual explainer
Loading lab
Explanation

A flattened patch is a vector of raw pixel values. Before the Transformer sees it, each patch is mapped to a D-dimensional embedding by one learned matrix. This single linear layer is the only part of ViT that looks at pixels directly.

A Trainable Linear Projection

The Transformer uses a constant latent vector size D through all of its layers. So the paper flattens the patches and maps them to D dimensions with a trainable linear projection E:

x_p^i E,    E ∈ ℝ^((P²·C)×D),    i = 1, …, N

The output of this projection is called the patch embeddings. The same E is applied to every patch, so the whole step is a single matrix multiplication of the (N × P²C) patch matrix by the (P²C × D) matrix E, producing an (N × D) sequence. In practice a bias is added too.

It Is a Convolution in Disguise

Look at what E does to one patch: each of the D output values is a weighted sum over all P·P·C pixels of the patch, with weights that do not depend on where the patch is. That is precisely a convolution with D filters of size P×P, applied with a stride of P so the windows tile the image without overlapping:

patch embedding  =  conv2d(C → D, kernel = P, stride = P)
E[(a, b, c), d]  =  W_conv[d, c, a, b]

The paper describes this case explicitly when discussing the hybrid model: the patch embedding projection E is applied to patches extracted from a CNN feature map, and in the special case of 1×1 patches it amounts to flattening the feature map and projecting it to D dimensions. Public implementations usually write the patch embedding as a strided convolution because it is fast and needs no explicit reshaping. The code cell builds both and checks they give the same tokens.

Parameter Counts

params(E) = P * P * C * D (+ D for the bias) model P D P*P*3*D ViT-B/32 32 768 2,359,296 ViT-B/16 16 768 589,824 ViT-L/16 16 1024 786,432 ViT-H/14 14 1280 752,640

These are tiny next to the encoder: ViT-B/16 has about 86 million parameters, of which the patch embedding is well under 1%. Notice the trade-off between the two B models. ViT-B/32 has four times the embedding weights of ViT-B/16 (bigger patches mean longer input vectors) but a sequence four times shorter.

What the Filters Learn

Since E is linear, each of its D columns is a filter over a patch and can be shown as a small image. The paper does this in Figure 7 (left) for ViT-L/32: the top principal components of the learned embedding filters resemble plausible basis functions for a low-dimensional representation of the fine structure within each patch. They look like smooth gradients, edges and stripes at various orientations and frequencies, much like the first layer of a CNN.

The code cell shows the idea with three hand-made filters instead of learned ones: the mean colour of a patch, a left-versus-right gradient and a top-versus-bottom gradient. Projecting all 196 patches onto them gives three 14×14 “feature maps”, one value per patch.

Linear, and Only Once

Everything finer than a patch must be captured by this one linear map. There is no nonlinearity between pixels and tokens, and after this step the model never sees pixels again. Spatial detail inside a patch can survive only in the D numbers of its embedding. That is one reason the patch size matters so much for accuracy: smaller patches mean less within-patch detail to compress into each token.

The key insight to carry forward: “cut into patches, flatten, multiply by E” and “a conv layer with kernel size and stride both equal to P” are the same operation. ViT's stem is a single, very large-stride convolution, followed by a model that has no convolutions at all.
Interactive code

Edit the code and press Ctrl + Enter to run it in your browser.

pynumpy · matplotlib
Exercises
1

ViT's patches do not overlap because stride = kernel = P. Run conv2d with kernel 16 and stride 8 instead. How many tokens does the image produce, and how many times is each interior pixel used?

2

In the hybrid model, patches are taken from a CNN feature map instead of raw pixels, and the paper notes they can be 1×1. Take a random 14 × 14 × 1024 feature map, patchify it with P = 1, and project it to D = 768. What are N and the size of E?

Key takeaway

Each flattened patch is multiplied by the same learned matrix E to give a D-dimensional patch embedding. This is exactly a strided convolution with kernel = stride = P, it holds well under 1% of ViT-Base's parameters, and it is the only place the model sees raw pixels.

Up next

Patch embeddings alone carry no position and no image summary. Next: the class token and E_pos.