Learn
Lesson 03 / 14Warm-up15 min

Similarity via Dot Products

The geometric foundation of every attention computation

By the end of this lesson you can

  • Compute dot products and understand them as cosine similarity
  • See why high dot product = high semantic relevance
  • Visualise vector similarity in 2D
Visual explainer
Loading lab
Explanation

At its heart, attention is just asking: "How similar is this query to each key?" The dot product is how we measure similarity - and understanding it geometrically is essential for understanding why attention works.

The Dot Product: A Refresher

Given two vectors a = [a₁, a₂, ..., aₙ] and b = [b₁, b₂, ..., bₙ], their dot product is:

a · b = a₁b₁ + a₂b₂ + ... + aₙbₙ = Σᵢ aᵢbᵢ

Geometrically, this equals:

a · b = |a| · |b| · cos(θ)

where θ is the angle between them. This formula is the key. When two vectors point in the same direction (θ = 0°), cos(θ) = 1 and the dot product is maximised. When they're perpendicular (θ = 90°), cos(θ) = 0. When they point in opposite directions (θ = 180°), cos(θ) = -1.

A Concrete Example

Imagine we have 3D embedding vectors (d_model = 3 for clarity):

"bank" (river): [0.8, 0.1, 0.1]
"bank" (finance): [0.1, 0.8, 0.1]
"river": [0.7, 0.2, 0.1]
"money": [0.1, 0.7, 0.2]

Dot product of "bank(river)" with "river": 0.8×0.7 + 0.1×0.2 + 0.1×0.1 = 0.56 + 0.02 + 0.01 = 0.59
Dot product of "bank(river)" with "money": 0.8×0.1 + 0.1×0.7 + 0.1×0.2 = 0.08 + 0.07 + 0.02 = 0.17

The "river" bank attends much more strongly to "river" than to "money" - exactly what we'd want for disambiguation. This is the entire mechanism of attention, in miniature.

Why Not Euclidean Distance?

You might wonder: why dot product and not L2 distance? Two reasons. First, dot products are a single matrix multiplication - one of the most hardware-optimised operations that exists. Second, cosine similarity (dot product of unit vectors) is scale-invariant, which is useful when embedding magnitudes vary.

The core intuition: In the attention mechanism, query vectors ask "what am I looking for?" and key vectors say "what do I contain?" The dot product Q·Kᵀ measures how well each query matches each key - it's a learned similarity score.
Interactive code

Edit the code and press Ctrl + Enter to run it in your browser.

pynumpy · matplotlib

Key takeaway

a·b = |a||b|cos(θ). Same-direction vectors → high dot product → strong attention. This is the entire scoring mechanism.

Up next

Now we combine dot products, softmax, and value vectors into the full attention formula.