The distributional hypothesis

Tokens and Embeddings, lesson 6

Words used in the same contexts end up with similar vectors.

You can score two vectors with cosine similarity ("Cosine similarity and the dot product"). This lesson shows where the numbers in an embedding come from.

Part 1. Count the neighbours

Here's a tiny pile of text, six sentences long:

"the cat sat on the mat", "the dog sat on the rug", "the cat ate the fish", "the dog ate the meat", "the car drove on the road", "the bus drove on the road".

For each word, write down the words right next to it, one on each side. Skip "the", because it sits next to everything and would make every word look alike.

Example 1.1

Rows of neighbour counts

cat sat on mat
dog sat on rug
cat ate fish
dog ate meat
car drove on road
bus drove on road
The sentences with "the" taken out.
catsat 1ate 1
dogsat 1ate 1
cardrove 1
Each word's neighbours, counted. Each row is a vector, with one slot per word in the vocabulary. Most slots are 0.
cosine(cat, dog)=1.0
cosine(cat, car)=0.0
Cat and dog have identical rows, so their cosine is 1. Cat and car share no neighbours, so theirs is 0.

Context window

How far either side of a word you look for neighbours. Here it's 1. Real systems use somewhere from 2 to 10. A small window tends to group words that do the same job (cat, dog); a wide one also pulls in words from the same topic (cat, fish).

Your turn

Cat and dog never appear in the same sentence in that pile. Why do their rows match anyway?

Nothing compares cat to dog directly. Each one only collects its own neighbours. They match because the same words keep turning up around both, and that's the whole idea behind learned embeddings.

Show the answer

C. Nothing compares cat to dog directly. Each one only collects its own neighbours. They match because the same words keep turning up around both, and that's the whole idea behind learned embeddings.

Distributional hypothesis

The idea that words used in the same contexts tend to mean similar things. The linguist J. R. Firth put it this way in 1957: "You shall know a word by the company it keeps." Every method in this lesson leans on it.

Part 2. Squeeze the rows

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come

Squeeze the rows
A guessing game
Watch it learn

See Pro plans I have Pro: sign in

Next lessonA tour of trained embedding spacesNeighbours, analogies and clusters in vectors a real model learned. Start lesson 7

Tokens and Embeddings overviewSpotted a mistake? Tell us.

06 · The distributional hypothesis