insideout Get Pro

07 · A tour of trained embedding spaces

Pro

Tokens and Embeddings, lesson 7

A tour of trained embedding spaces

Neighbours, analogies and clusters in vectors a real model learned.

You know where embeddings come from ("The distributional hypothesis") and how cosine similarity scores them ("Cosine similarity and the dot product").

Part 1. Words as points

Trained word vectors have hundreds of numbers each, far too many to draw. To look at them, people squash each vector down to two numbers and plot the words on a flat map. The map below is in that spirit, but laid out by hand. Click a word to ring the three words nearest to it.

A hand-made word mapLive

Groups appear: animals in one corner, food in another, vehicles and feelings in two more. Maps of real trained vectors show the same kind of clusters. Say a model learned that reviews mentioning "dog" are usually happy. If "puppy" sits right next to "dog", the model gets a head start on "puppy", even if it barely saw that word in training.

On a flat map, "near" can be measured with a ruler: how far across, how far up, then Pythagoras.

Example 1.1

Two distances, by hand

  1. cat (2.1, 7.6)dog (2.9, 7.9)

    Two points from the map.

  2. across 0.8up 0.3
    √(0.8² + 0.3²)=√0.73=0.85

    Square both gaps, add, take the square root. Cat and dog are 0.85 apart.

  3. cat (2.1, 7.6)car (7.2, 2.0)
    √(5.1² + 5.6²)=√57.37=7.57

    Cat and car are 7.57 apart: nearly nine times as far. Similar meaning, small distance.

Your turn

On the map, happy is at (2.4, 2.9), glad at (1.8, 2.4), sad at (3.6, 1.6) and cat at (2.1, 7.6). Which word is closest to happy?

glad is √(0.6² + 0.5²) ≈ 0.78 away, sad is √(1.2² + 1.3²) ≈ 1.77, cat ≈ 4.71. Notice sad is still fairly close: it shows up in the same kind of sentences as happy. "Blind spots of cosine similarity" comes back to that.

On the full vectors, with hundreds of numbers each, the rest of this lesson uses cosine instead of the ruler, for the reasons in "Cosine similarity and the dot product".

Part 2. Neighbours in real vectors

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. Neighbours in real vectors
  2. Analogies, and the fine print
  3. The text's habits come along
  4. One point per word
See Pro plansI have Pro: sign in
Next lessonBlind spots of cosine similarityOpposites, negation and rare words that similarity scores get wrong. Start lesson 8

Tokens and Embeddings overviewSpotted a mistake? Tell us.