Why Transformers for Images?
CNN inductive biases vs a general-purpose architecture, and why scale changes the answer
By the end of this lesson you can
- Name the inductive biases of a CNN: locality, 2D neighbourhood structure and translation equivariance
- Summarise prior attempts at attention for images and why they needed special engineering
- State the paper's thesis that large-scale training trumps inductive bias
By 2020 the Transformer had taken over natural language processing, while computer vision still belonged to convolutional networks. The Vision Transformer (ViT) paper asks a blunt question: what happens if you take a standard Transformer, change as little as possible, and feed it an image? The answer turns out to depend almost entirely on how much data you have.
What a Convolution Assumes
Every architecture makes assumptions about its data before it sees a single example. These assumptions are called inductive biases. Section 3.1 of the paper lists the ones a CNN has baked into every layer:
- Locality. A 3×3 filter only looks at a pixel and its immediate neighbours. Distant pixels can only interact after many layers.
- Two-dimensional neighbourhood structure. The filter is laid out on the 2D grid, so the network knows which pixels are above, below, left and right of each other.
- Translation equivariance. The same filter slides over every position, so shifting the input shifts the output by the same amount. A cat detector learned in the top-left corner works in the bottom-right.
These are good assumptions for natural images, and they make CNNs very data-efficient: the network does not have to learn from examples that an edge is an edge wherever it appears.
What a Transformer Assumes
Almost nothing. A Transformer layer treats its input as a set of tokens. Self-attention lets every token interact with every other token, with weights computed from the content itself. The paper is explicit about how little image-specific structure survives in ViT: only the MLP layers are local and translationally equivariant, while the self-attention layers are global. The 2D neighbourhood structure is used very sparingly: at the start, when the image is cut into patches, and at fine-tuning time, when position embeddings are adjusted for a new resolution. Position embeddings at initialisation carry no information about the 2D positions of the patches, so all spatial relations between patches have to be learned from scratch.
Earlier Attempts
Applying self-attention to images was not new, but naive self-attention over pixels is hopeless: a 224×224 image has 50,176 pixels, and every pixel attending to every other pixel costs quadratic time and memory. The related work section surveys the workarounds:
- Local attention (Parmar et al., 2018): each query pixel attends only to a local neighbourhood, which reintroduces locality by hand.
- Sparse and blockwise attention (Sparse Transformers, Child et al., 2019; Weissenborn et al., 2019): scalable approximations to global attention.
- Axial attention (Ho et al., 2019; Wang et al., 2020): attention along one axis at a time, rows then columns.
- Hybrids that add attention on top of, or inside, a CNN backbone.
Many of these specialised designs showed promising results, but the paper notes that they require complex engineering to run efficiently on hardware accelerators. The closest prior work is Cordonnier et al. (2020), who extract 2×2 patches and apply full self-attention on top. That is essentially the ViT recipe, but 2×2 patches restrict the model to small-resolution images, and that work did not show that the approach could compete with state-of-the-art CNNs.
The Thesis: Scale Trumps Inductive Bias
ViT deliberately keeps the standard Transformer and its efficient implementations. The interesting finding is what happens as the pre-training set grows. Trained on a mid-sized dataset such as ImageNet without strong regularisation, ViT reaches accuracies a few percentage points below ResNets of comparable size. The paper reads this as expected: without the right inductive biases, a model does not generalise well from insufficient data.
The picture changes when the models are pre-trained on larger datasets: ImageNet-21k (about 14 million images) and JFT-300M (about 300 million). In the authors' words, “large scale training trumps inductive bias”. Pre-trained on JFT-300M, the best ViT reaches 88.55% top-1 on ImageNet and matches or beats state-of-the-art CNNs on many benchmarks, while needing substantially fewer computational resources to pre-train.
A Toy Version of the Trade-off
The code cell makes this concrete with a 1D “image”. The true input-output map is mostly a convolution plus a small part that depends on position. One model can only learn a 5-tap convolution (a strong inductive bias, 5 parameters). The other can learn any linear map (no bias, 1,024 parameters). With little data the convolution wins easily. With enough data the flexible model overtakes it, because the strong prior is slightly wrong and no amount of data can fix that. The lab above lets you explore the same crossover.
Edit the code and press Ctrl + Enter to run it in your browser.
How wrong does the convolutional assumption have to be before data can overcome it? For non-equivariant strengths 0, 0.1, 0.3 and 1.0, find the smallest training size in `sizes` at which the dense model beats the conv model.
The matrices that commute with every circular shift are exactly the circulant ones (convolutions). Project the random dense map M onto circulant matrices by averaging each wrapped diagonal, and check that the projection is shift-equivariant for every shift while M is not.
Key takeaway
CNNs build in strong assumptions about images, which makes them data-efficient. ViT drops almost all of them. On ImageNet alone it trails comparable ResNets, but pre-trained on 14M to 300M images it matches or beats them: large-scale training trumps inductive bias.
Up next
To use a standard Transformer, the image must first become a sequence of tokens.