Similarity via Dot Products
The geometric foundation of every attention computation
By the end of this lesson you can
- Compute dot products and understand them as cosine similarity
- See why high dot product = high semantic relevance
- Visualise vector similarity in 2D
At its heart, attention is just asking: "How similar is this query to each key?" The dot product is how we measure similarity - and understanding it geometrically is essential for understanding why attention works.
The Dot Product: A Refresher
Given two vectors a = [a₁, a₂, ..., aₙ] and b = [b₁, b₂, ..., bₙ], their dot product is:
Geometrically, this equals:
where θ is the angle between them. This formula is the key. When two vectors point in the same direction (θ = 0°), cos(θ) = 1 and the dot product is maximised. When they're perpendicular (θ = 90°), cos(θ) = 0. When they point in opposite directions (θ = 180°), cos(θ) = -1.
A Concrete Example
Imagine we have 3D embedding vectors (d_model = 3 for clarity):
"bank" (finance): [0.1, 0.8, 0.1]
"river": [0.7, 0.2, 0.1]
"money": [0.1, 0.7, 0.2]
Dot product of "bank(river)" with "river": 0.8×0.7 + 0.1×0.2 + 0.1×0.1 = 0.56 + 0.02 + 0.01 = 0.59
Dot product of "bank(river)" with "money": 0.8×0.1 + 0.1×0.7 + 0.1×0.2 = 0.08 + 0.07 + 0.02 = 0.17
The "river" bank attends much more strongly to "river" than to "money" - exactly what we'd want for disambiguation. This is the entire mechanism of attention, in miniature.
Why Not Euclidean Distance?
You might wonder: why dot product and not L2 distance? Two reasons. First, dot products are a single matrix multiplication - one of the most hardware-optimised operations that exists. Second, cosine similarity (dot product of unit vectors) is scale-invariant, which is useful when embedding magnitudes vary.
Edit the code and press Ctrl + Enter to run it in your browser.
Key takeaway
a·b = |a||b|cos(θ). Same-direction vectors → high dot product → strong attention. This is the entire scoring mechanism.
Up next
Now we combine dot products, softmax, and value vectors into the full attention formula.