Learn
Lesson 05 / 14Core20 min

Queries, Keys, and Values

The library analogy and why separate projections matter

By the end of this lesson you can

  • Understand what W_Q, W_K, W_V matrices learn to do
  • See why Q≠K (asymmetric attention) is important
  • Know why V is separate from K
Visual explainer
Loading lab
Explanation

The Q, K, V projections are what make attention learnable. Without them, every token would query and answer with the same representation. The projections let the model learn different "roles" for each token depending on context.

The Library Analogy

Imagine a library. You walk in with a search query (Q) - "I want books about gradient descent." Every book in the library has an index card (K) describing its contents - "This book covers optimisation algorithms." The library uses your query to match against all index cards, finds the most relevant books, and hands you their content (V).

The weight matrices W_Q, W_K, W_V are learned transformations that convert the raw token representation into specialised query, key, and value roles. During training, the model learns what kinds of queries are useful, what keys should advertise, and what values should carry.

The Projection Matrices

In the paper (d_model=512, d_k=d_v=64 for each head):

W_Q: [512 × 64] - projects input to query space
W_K: [512 × 64] - projects input to key space
W_V: [512 × 64] - projects input to value space

These matrices are not shared between heads or layers - each attention head in each layer has its own W_Q, W_K, W_V. This is where most of the model's parameters live.

Why Separate Q and K?

If Q = K (same projection), the attention matrix QKᵀ is symmetric: token i attending to token j would always equal token j attending to token i. But language isn't symmetric. "The pronoun refers to the noun" is directional. Using separate W_Q and W_K lets the model learn asymmetric attention patterns.

Why Separate V from K?

What information a token matches on (its key) doesn't have to be the same as what it provides (its value). A token like "bank" might have a key that signals "I'm a noun near water" but a value that contains full contextual information about financial institutions. Separating K and V gives the model this flexibility.

Think of it this way: W_Q teaches each token "how to ask questions," W_K teaches it "how to describe itself for matching," and W_V teaches it "what information to share when selected." Three different roles, three different learned projections.
Interactive code

Edit the code and press Ctrl + Enter to run it in your browser.

pynumpy · matplotlib

Key takeaway

W_Q, W_K, W_V are learned. They give each token three different roles: what to ask for, how to describe itself, and what to share when selected.

Up next

Apply QKV to a full sequence - that's self-attention.