Scaled Dot-Product Attention
Attention(Q,K,V) = softmax(QKᵀ/√d_k)·V - derived from scratch
By the end of this lesson you can
- Derive every term in the attention formula
- Understand why √d_k scaling prevents softmax saturation
- Implement scaled dot-product attention in numpy
This is the heart of the paper. Everything else - multi-head attention, the encoder, the decoder - is built on top of this one formula. Understand it deeply and the rest follows naturally.
The Formula
Let's unpack every symbol before we derive the whole thing from scratch.
The Inputs: Q, K, V
Given a sequence of n tokens, each represented as a d-dimensional vector, we produce three matrices by projecting the input with learned weight matrices:
K = X · W_K shape: [n × d_k]
V = X · W_V shape: [n × d_v]
Q (Query): "What am I looking for?"
K (Key): "What do I contain?"
V (Value): "What information do I actually carry?"
Step 1: Compute Raw Attention Scores - QKᵀ
QKᵀ produces an [n × n] matrix. Entry [i, j] is the dot product between query i and key j - a raw score of "how much should token i attend to token j?"
For n=4, Q=[4×8] and K=[4×8], so QKᵀ is [4×4] - one score per (query, key) pair.
Step 2: Scale by 1/√d_k
If d_k is large, the dot products grow large in magnitude. Why? In expectation, each element of Q and K has variance 1 (after good initialisation). The dot product of two d_k-dimensional vectors then has variance d_k (it's a sum of d_k unit-variance terms). Large values push softmax into regions where gradients vanish.
Dividing by √d_k brings the variance back to 1: Var(Q·K / √d_k) = Var(Q·K) / d_k = d_k / d_k = 1. Problem solved.
Step 3: Softmax - Convert Scores to Weights
Softmax converts the raw scores into a probability distribution over positions:
Each row of the score matrix becomes a probability distribution summing to 1. The highest score gets the most weight; low scores become near-zero. This is the "soft" in attention: instead of selecting one token sharply, we get a weighted combination.
Step 4: Weighted Sum of Values - ·V
Finally, we multiply the attention weights by V. For each query token i, we compute:
This is a weighted average of all value vectors, where the weights came from how much the query matched each key. The output is the "answer" to the query - a blend of information from all tokens, weighted by relevance.
Edit the code and press Ctrl + Enter to run it in your browser.
Try removing the 1/√d_k scaling. With d_k=64, what happens to the attention weights distribution?
Key takeaway
The formula has four steps: score (QKᵀ), scale (÷√d_k), normalise (softmax), aggregate (·V). Each step has a specific, necessary purpose.
Up next
The formula uses Q, K, V - but where do they come from? Learned projections.