insideout Get Pro

04 · Looking up a token's vector

Pro

Tokens and Embeddings, lesson 4

Looking up a token's vector

An embedding table turns each token ID into a learned vector.

You know text becomes a list of token IDs ("Training a BPE tokenizer") and that a layer multiplies its inputs by weights ("Stacking neurons into layers").

Part 1. An ID is a label, not an amount

The tokenizer hands the network a list of IDs. In GPT-2, "Hello, world!" is 15496, 11, 995, 0. The network's layers do weighted sums. Can the IDs go straight in?

Your turn

Say you fed the raw IDs into a weighted sum. " world" is 995 and the comma is 11. What would the network wrongly assume?

A weighted sum treats a bigger input as more of something. But IDs are just positions in a list, set by the order the merges happened in. Token 994 has nothing to do with token 995, and 995 isn't "more" than 11.

So each ID is used to look up a row in a table instead. Each row is a list of numbers, and those numbers are parameters: they start random and get tuned during training like every other weight.

Example 1.1

Looking up "the cat"

  1. thecat
    →
    id 0id 2

    The tokenizer turns the text into IDs. In this toy vocabulary, "the" is 0 and "cat" is 2.

  2. row 0[0.3, −0.1, 0.5, 0.2]
    row 1[0.0, 0.7, −0.4, 0.1]
    row 2[0.1, −0.4, 0.8, 0.0]
    row 3[−0.6, 0.2, 0.1, 0.9]

    The table has one row per token in the vocabulary. Here each row holds 4 numbers. Real models use thousands.

  3. the → [0.3, −0.1, 0.5, 0.2]
    cat → [0.1, −0.4, 0.8, 0.0]

    Each ID picks out its row. The network never sees the IDs, only these lists of numbers.

Embedding

The row of numbers for a token is its embedding, and the table is the embedding table (or embedding layer). It has one row per token in the vocabulary.

Part 2. A lookup is a layer

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. A lookup is a layer
  2. How big the table gets
See Pro plansI have Pro: sign in
Next lessonCosine similarity and the dot productTwo ways to score how close two vectors are, and when they disagree. Start lesson 5

Tokens and Embeddings overviewSpotted a mistake? Tell us.