Input Representation: [CLS], [SEP] and Three Embeddings
Token + segment + position, packed into one sequence
By the end of this lesson you can
- Pack a sentence pair with [CLS] and [SEP]
- Explain the roles of token, segment and position embeddings
- Apply padding, attention masks and longest-first truncation
Many tasks involve two pieces of text: a question and a passage, a premise and a hypothesis. BERT handles all of them with one flat input sequence and three embeddings added together.
Packing the Sequence
Every input starts with a special [CLS] token. A single sentence ends with [SEP]; a pair is written as:
The final hidden vector of [CLS], written C, is used as the summary of the whole input for classification. It has no meaning of its own until pre-training (next sentence prediction) and fine-tuning give it one.
Three Embeddings
The vector for each position is a sum:
Token embeddings are the 30,000 × H WordPiece table. Segment embeddings are two learned vectors, E_A and E_B, telling the model which sentence a token belongs to. Position embeddings are learned, one per position up to 512. Unlike the original Transformer's fixed sinusoids, BERT treats positions as ordinary parameters. The sum is layer-normalised before the first encoder layer.
Padding, Masks and Truncation
Batches need equal lengths, so short inputs are padded with [PAD] and an attention mask marks which positions are real. Padding keys get a large negative score before the softmax, so no real token attends to them. Pairs that are too long are trimmed one token at a time from the end of whichever sentence is currently longer.
Edit the code and press Ctrl + Enter to run it in your browser.
Write truncate_pair(a, b, max_len) implementing BERT's longest-first rule, then pack a 10-token A with a 3-token B into max_len = 10.
Key takeaway
Every BERT input is [CLS] A [SEP] (B [SEP]). Each position's vector is E_token + E_segment + E_position, layer-normalised. One format serves every task.
Up next
Those vectors enter the encoder. How big is it, and how does it differ from GPT?