insideout Get Pro

08 · Blind spots of cosine similarity

Pro

Tokens and Embeddings, lesson 8

Blind spots of cosine similarity

Opposites, negation and rare words that similarity scores get wrong.

You know embeddings come from the company words keep ("The distributional hypothesis") and how cosine similarity scores them ("Cosine similarity and the dot product").

Part 1. Opposites keep the same company

Fill the gap: "the soup is too ___", "a ___ drink", "___ weather", "the water was ___". Hot fits every one. So does cold.

Embeddings learn from exactly those gaps. Opposites like hot and cold, good and bad, buy and sell are used in the same kinds of sentences, so they collect the same neighbours.

Example 1.1

Hot, cold and spicy

  1. neighbours:soupdrinkweatherwaterday
    hot32241
    cold23241
    spicy31000

    Made-up neighbour counts, like the tables in "The distributional hypothesis". Hot and cold share almost every neighbour. Spicy only shows up around food.

  2. hot · cold=6 + 6 + 4 + 16 + 1=33
    33 ÷ (√34 × √34)=0.97

    Cosine of hot and cold: 0.97, nearly the top of the scale.

  3. hot · spicy=9 + 2=11
    11 ÷ (√34 × √10)=0.60

    Hot and spicy, which can mean the same thing about food, score only 0.60. By this measure, hot is far more like its opposite.

Your turn

In the Google News word2vec vectors, which of these sits among the very closest words to "good"?

"bad" is right near the top, next to "great" and "decent". Good and bad fill the same gaps ("that was a ___ idea"). The vectors measure how words are used, and opposites are used in the same places. "evil" belongs to a narrower set of sentences.

So a high cosine means "used in similar places". Often that's the same as "means something similar". For opposites it isn't, and the vectors can't tell the difference on their own.

Part 2. "not" barely moves the vector

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. "not" barely moves the vector
  2. Close is not the same as correct
See Pro plansI have Pro: sign in
Next lessonSemantic search and clusteringUse embeddings to search, group and label text. Start lesson 9

Tokens and Embeddings overviewSpotted a mistake? Tell us.