insideout Get Pro

09 · Semantic search and clustering

Pro

Tokens and Embeddings, lesson 9

Semantic search and clustering

Use embeddings to search, group and label text.

You can score two vectors with cosine similarity ("Cosine similarity and the dot product") and you know where that score misleads ("Blind spots of cosine similarity").

Part 1. Search: rank by cosine

A help centre has five articles. Someone types "I forgot my login". None of the article titles contain those words, yet the right one is obvious to you: "Reset your password".

Embeddings get there the same way you compared words. Turn every article into a vector once and store it. When a question comes in, turn it into a vector too, score it against every stored vector with cosine similarity, and sort.

Example 1.1

One search, start to finish

  1. Reset your password(0.9, 0.1, 0.0)
    Change your email address(0.6, 0.5, 0.1)
    Update your card(0.1, 0.9, 0.2)
    Download an invoice(0.0, 0.7, 0.6)
    Cancel your plan(0.2, 0.4, 0.8)

    The articles, embedded ahead of time. Three made-up numbers each; a real model gives hundreds.

  2. "I forgot my login"→(0.8, 0.2, 0.1)

    The question goes through the same embedding model.

  3. Reset your password 0.98
    Change your email address 0.90
    Cancel your plan 0.42
    Update your card 0.36
    Download an invoice 0.26

    Score and sort. The password article wins without sharing a single word with the question.

Semantic search

Finding text by closeness of meaning (embedding similarity) instead of by shared words. The stored vectors are often called the index.

Your turn

The articles were embedded with model A. You upgrade and embed new questions with model B. What happens to the search?

Each model learns its own map. Number 1 in A's vectors has no connection to number 1 in B's, and the two may not even have the same length. Questions and documents must go through the same model, so switching models means embedding everything again.

Scoring every stored vector works fine for thousands of documents. At millions, systems use a special index that finds the near ones without checking every vector, at the cost of sometimes missing one.

Part 2. Classification: the nearest average

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. Classification: the nearest average
  2. Clustering: let the groups appear
See Pro plansI have Pro: sign in
Next lessonA bigram language modelPredict the next character by counting, then by training, and compare the two. Start lesson 10

Tokens and Embeddings overviewSpotted a mistake? Tell us.