Queries, Keys, and Values
The library analogy and why separate projections matter
By the end of this lesson you can
- Understand what W_Q, W_K, W_V matrices learn to do
- See why Q≠K (asymmetric attention) is important
- Know why V is separate from K
The Q, K, V projections are what make attention learnable. Without them, every token would query and answer with the same representation. The projections let the model learn different "roles" for each token depending on context.
The Library Analogy
Imagine a library. You walk in with a search query (Q) - "I want books about gradient descent." Every book in the library has an index card (K) describing its contents - "This book covers optimisation algorithms." The library uses your query to match against all index cards, finds the most relevant books, and hands you their content (V).
The weight matrices W_Q, W_K, W_V are learned transformations that convert the raw token representation into specialised query, key, and value roles. During training, the model learns what kinds of queries are useful, what keys should advertise, and what values should carry.
The Projection Matrices
In the paper (d_model=512, d_k=d_v=64 for each head):
W_K: [512 × 64] - projects input to key space
W_V: [512 × 64] - projects input to value space
These matrices are not shared between heads or layers - each attention head in each layer has its own W_Q, W_K, W_V. This is where most of the model's parameters live.
Why Separate Q and K?
If Q = K (same projection), the attention matrix QKᵀ is symmetric: token i attending to token j would always equal token j attending to token i. But language isn't symmetric. "The pronoun refers to the noun" is directional. Using separate W_Q and W_K lets the model learn asymmetric attention patterns.
Why Separate V from K?
What information a token matches on (its key) doesn't have to be the same as what it provides (its value). A token like "bank" might have a key that signals "I'm a noun near water" but a value that contains full contextual information about financial institutions. Separating K and V gives the model this flexibility.
Edit the code and press Ctrl + Enter to run it in your browser.
Key takeaway
W_Q, W_K, W_V are learned. They give each token three different roles: what to ask for, how to describe itself, and what to share when selected.
Up next
Apply QKV to a full sequence - that's self-attention.