insideout Get Pro

07 · Encoding word order and RoPE

Pro

Transformers, lesson 7

Encoding word order and RoPE

Attention ignores order unless positions are added, or rotated in.

You know a token ID picks out one row of an embedding table ("Looking up a token's vector"), and how attention scores every token against every other ("Attention as a soft lookup").

Part 1. Order goes missing

"dog bites man" and "man bites dog" use the same three tokens. Look each one up in the embedding table and you get the same three vectors. Only the order differs.

Now think back to attention. Every query is dotted with every key, and the weights blend every value. Nothing in that recipe asks where a vector sits in the list. Swap two words and each one gets the same scores against the same partners, only in a different order.

Example 1.1

Two sentences, one bag of vectors

  1. dogbitesman
    (0.4, 0.1)(−0.3, 0.5)(0.2, −0.6)

    The first sentence, looked up in a toy table with two numbers per token.

  2. manbitesdog
    (0.2, −0.6)(−0.3, 0.5)(0.4, 0.1)

    The second sentence. Same three vectors, shuffled.

  3. {(0.4, 0.1), (−0.3, 0.5), (0.2, −0.6)}

    To anything that ignores the order of the list, the two are the same input. Who bit whom is lost.

Your turn

A model sees only token vectors, and its comparisons ignore list order. What does it get for "dog" in the two sentences?

The lookup only knows the token ID, and the ID for dog is the same in both. With nothing else added, the model has no way to tell the first word from the last. Something has to carry the position in.

Part 2. Add a vector for each position

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. Add a vector for each position
  2. Rotating queries and keys
  3. Fast and slow dials
See Pro plansI have Pro: sign in
Next lessonNorm, attention, MLP, residualThe four parts of one transformer block, and why each is there. Start lesson 8

Transformers overviewSpotted a mistake? Tell us.