Words to Vectors: Token Embeddings
Tokenisation, embedding matrices, and what d_model = 512 actually means
By the end of this lesson you can
- Understand byte-pair encoding tokenisation
- See how the embedding matrix maps token IDs to vectors
- Know why embeddings are scaled by √d_model before positional encoding
Before tokens can attend to each other, they need to exist as vectors. The embedding layer is where words enter the mathematical world.
From Text to Tokens
A neural network can't process the word "cat" directly. It needs numbers. The first step is tokenisation: splitting text into sub-word units. The Attention paper used byte-pair encoding (BPE) with a vocabulary of ~37,000 tokens. Common words like "the" map to a single token; rare words are split into pieces ("transformer" → "transform" + "er").
Each token gets assigned an integer ID. "The" might be 464, "cat" might be 3797, and so on. This gives us a sequence of integers: [464, 3797, 3332, 319, 262, 2603].
The Embedding Matrix
The embedding layer is a lookup table: a matrix E of shape [vocab_size × d_model]. In the Attention paper, vocab_size ≈ 37,000 and d_model = 512. So E has about 19 million parameters - roughly 10% of the whole model.
For each token ID i, the embedding is simply row i of E: a vector of 512 floats. These floats start random and are learned during training. After training on enough data, similar words end up with similar vectors:
cosine_similarity("king", "potato") ≈ 0.08
What d_model = 512 Actually Means
Each token is represented as a point in 512-dimensional space. You can think of each dimension as a "feature axis" - though in practice the dimensions don't have clean human-interpretable meanings. What matters is that the geometry of this 512D space encodes semantic relationships.
The famous word2vec result holds here too: E["king"] - E["man"] + E["woman"] ≈ E["queen"]. The embedding space has arithmetic structure that reflects real-world relationships.
Scaling by √d_model
In the paper, the authors scale the embedding vectors by √d_model before adding positional encodings. This is because positional encodings have a fixed scale, while embedding vectors can grow with d_model. Scaling ensures the two are on the same magnitude. It's a small detail that matters in practice.
Edit the code and press Ctrl + Enter to run it in your browser.
Key takeaway
Each token is a learned 512-dimensional vector. The embedding matrix (vocab × d_model) is the model's vocabulary in geometric form.
Up next
Next: the mathematical tool that powers attention - the dot product.