What trained heads learn

Transformers, lesson 6

Previous-token, induction and other patterns found in real models.

This builds on "Heads in parallel": each layer runs many attention heads side by side, and each head has its own row of weights for every token.

Part 1. Reading an attention pattern

Researchers can run a trained model on some text and record every head's weights. For one head, that gives a grid: one row per token doing the looking, one column per token being looked at. Because of the causal mask, everything above the diagonal is zero.

Example 1.1

One head's weights on "the cat sat on the mat"

cat:the 0.92cat 0.08
Read one row at a time. The row for "cat" says where "cat" took its information from: almost all of it came from "the".
sat:the 0.03cat 0.90sat 0.07
The row for "sat" puts its weight on "cat". Again, the word right before it.
on:sat 0.88
the:on 0.91
mat:the 0.89
Every row does the same thing, whatever the words are. On the grid, that draws a bright stripe one step left of the diagonal. (These weights are illustrative.)

Attention pattern

The grid of one head's weights over a piece of text. Reading patterns is the most common first step when people try to work out what a head does.

Your turn

Which pattern would you expect from a head whose job is "copy in the word right before me"?

Row i puts its weight on column i − 1, which is a stripe just below the diagonal. That's a previous-token head. They're among the easiest heads to spot, and researchers have found them in many trained models, from tiny ones to large ones.

Show the answer

C. Row i puts its weight on column i − 1, which is a stripe just below the diagonal. That's a previous-token head. They're among the easiest heads to spot, and researchers have found them in many trained models, from tiny ones to large ones.

Part 2. Heads that continue a pattern

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come

Heads that continue a pattern
Heads that park on the first token
What patterns don't tell you

See Pro plans I have Pro: sign in

Next lessonEncoding word order and RoPEAttention ignores order unless positions are added, or rotated in. Start lesson 7

Transformers overviewSpotted a mistake? Tell us.

06 · What trained heads learn