Learn
Lesson 04 / 14Core25 min

Scaled Dot-Product Attention

Attention(Q,K,V) = softmax(QKᵀ/√d_k)·V - derived from scratch

By the end of this lesson you can

  • Derive every term in the attention formula
  • Understand why √d_k scaling prevents softmax saturation
  • Implement scaled dot-product attention in numpy
Visual explainer
Loading lab
Explanation

This is the heart of the paper. Everything else - multi-head attention, the encoder, the decoder - is built on top of this one formula. Understand it deeply and the rest follows naturally.

The Formula

Attention(Q, K, V) = softmax( QKᵀ / √d_k ) · V

Let's unpack every symbol before we derive the whole thing from scratch.

The Inputs: Q, K, V

Given a sequence of n tokens, each represented as a d-dimensional vector, we produce three matrices by projecting the input with learned weight matrices:

Q = X · W_Q   shape: [n × d_k]
K = X · W_K   shape: [n × d_k]
V = X · W_V   shape: [n × d_v]

Q (Query): "What am I looking for?"
K (Key): "What do I contain?"
V (Value): "What information do I actually carry?"

Step 1: Compute Raw Attention Scores - QKᵀ

QKᵀ produces an [n × n] matrix. Entry [i, j] is the dot product between query i and key j - a raw score of "how much should token i attend to token j?"

For n=4, Q=[4×8] and K=[4×8], so QKᵀ is [4×4] - one score per (query, key) pair.

Step 2: Scale by 1/√d_k

If d_k is large, the dot products grow large in magnitude. Why? In expectation, each element of Q and K has variance 1 (after good initialisation). The dot product of two d_k-dimensional vectors then has variance d_k (it's a sum of d_k unit-variance terms). Large values push softmax into regions where gradients vanish.

Dividing by √d_k brings the variance back to 1: Var(Q·K / √d_k) = Var(Q·K) / d_k = d_k / d_k = 1. Problem solved.

Step 3: Softmax - Convert Scores to Weights

Softmax converts the raw scores into a probability distribution over positions:

softmax(x)ᵢ = exp(xᵢ) / Σⱼ exp(xⱼ)

Each row of the score matrix becomes a probability distribution summing to 1. The highest score gets the most weight; low scores become near-zero. This is the "soft" in attention: instead of selecting one token sharply, we get a weighted combination.

Step 4: Weighted Sum of Values - ·V

Finally, we multiply the attention weights by V. For each query token i, we compute:

output[i] = Σⱼ attention_weight[i,j] · V[j]

This is a weighted average of all value vectors, where the weights came from how much the query matched each key. The output is the "answer" to the query - a blend of information from all tokens, weighted by relevance.

Common mistake: Forgetting the √d_k scaling. Without it, for large d_k (like 64), the softmax input has very large values, producing extremely peaked distributions with near-zero gradients. Training stalls.
Interactive code

Edit the code and press Ctrl + Enter to run it in your browser.

pynumpy · matplotlib
Exercises
1

Try removing the 1/√d_k scaling. With d_k=64, what happens to the attention weights distribution?

Key takeaway

The formula has four steps: score (QKᵀ), scale (÷√d_k), normalise (softmax), aggregate (·V). Each step has a specific, necessary purpose.

Up next

The formula uses Q, K, V - but where do they come from? Learned projections.