Learn
Lesson 03 / 13Warm-up20 min

Input Representation: [CLS], [SEP] and Three Embeddings

Token + segment + position, packed into one sequence

By the end of this lesson you can

  • Pack a sentence pair with [CLS] and [SEP]
  • Explain the roles of token, segment and position embeddings
  • Apply padding, attention masks and longest-first truncation
Visual explainer
Loading lab
Explanation

Many tasks involve two pieces of text: a question and a passage, a premise and a hypothesis. BERT handles all of them with one flat input sequence and three embeddings added together.

Packing the Sequence

Every input starts with a special [CLS] token. A single sentence ends with [SEP]; a pair is written as:

[CLS] my dog is cute [SEP] he likes play ##ing [SEP]

The final hidden vector of [CLS], written C, is used as the summary of the whole input for classification. It has no meaning of its own until pre-training (next sentence prediction) and fine-tuning give it one.

Three Embeddings

The vector for each position is a sum:

input_i = E_token[token_i] + E_segment[A or B] + E_position[i]

Token embeddings are the 30,000 × H WordPiece table. Segment embeddings are two learned vectors, E_A and E_B, telling the model which sentence a token belongs to. Position embeddings are learned, one per position up to 512. Unlike the original Transformer's fixed sinusoids, BERT treats positions as ordinary parameters. The sum is layer-normalised before the first encoder layer.

Padding, Masks and Truncation

Batches need equal lengths, so short inputs are padded with [PAD] and an attention mask marks which positions are real. Padding keys get a large negative score before the softmax, so no real token attends to them. Pairs that are too long are trimmed one token at a time from the end of whichever sentence is currently longer.

Why this design matters: one input format covers single sentences, sentence pairs and question plus passage. That is what lets the same pre-trained network be fine-tuned for every task without architectural changes.
Interactive code

Edit the code and press Ctrl + Enter to run it in your browser.

pynumpy · matplotlib
Exercises
1

Write truncate_pair(a, b, max_len) implementing BERT's longest-first rule, then pack a 10-token A with a 3-token B into max_len = 10.

Key takeaway

Every BERT input is [CLS] A [SEP] (B [SEP]). Each position's vector is E_token + E_segment + E_position, layer-normalised. One format serves every task.

Up next

Those vectors enter the encoder. How big is it, and how does it differ from GPT?