Transformers, lesson 5
Heads in parallel
Several smaller attention heads, each tracking a different relationship.
You should be comfortable with "Why the scores are scaled" and "Masking future tokens": queries score keys, softmax turns the scores into weights, and the weights blend the values.
Part 1. One head can only blend
Take the text "The animal ran because it". The model is about to guess the word after "it", and the vector for "it" wants two different things from earlier words. It wants to know which noun "it" points back to ("animal"). It also wants to know the word right before it ("because"), since that shapes what comes next.
Attention as you've seen it so far gives "it" one row of weights, and that row has to add up to 1. Watch what happens when one row tries to serve both needs.
Example 1.1
One row of weights, two jobs
- animal → [1, 0]because → [0, 1]
Two value vectors, kept tiny so you can follow them. The first number stands for "who", the second for "what came just before".
- animal 0.5because 0.5others 0
One head has to split its weight. Half goes to each word it cares about.
- 0.5 × [1, 0]+0.5 × [0, 1]=[0.5, 0.5]
The output is an average. Both signals arrive at half strength, mixed into one vector.
Your turn
A head puts weight 0.9 on "animal" and 0.1 on "because", using the same value vectors. What does "it" receive?
Part 2. Split, attend, join
Keep going with Pro
You've done the first part. Pro opens the rest of this lesson, and every other one.
- Split, attend, join
- More heads, same size
Transformers overviewSpotted a mistake? Tell us.