Search and Retrieval, lesson 2
Vector search
Find text by meaning with embeddings and nearest neighbours.
Keyword search scored both refund passages 0 for "can I get my money back", because they share no words with it. This lesson searches by meaning instead: turn every text into a list of numbers and look for the closest. You need BM25 from "Keyword search with TF-IDF and BM25" and cosine similarity from "Cosine similarity and the dot product".
By the end, you can:
Part 1. Meaning as a direction
Score how close two texts are with the cosine of their embeddings.
An embedding model turns any text into a list of numbers, trained so that texts with similar meaning get lists that point in similar directions. Real models give hundreds or thousands of numbers. Two are enough to see the idea: say the first measures how much a text is about money, the second how much it's about devices.
Example 1.1
A question and a refund passage
The question "how do I get my money back?" embeds to (1.2, 0.5). The passage "Enterprise plan: a full refund within 60 days of purchase." embeds to (2.4, 0.7). How close are they?
- Multiply and add1.2 × 2.4 + 0.5 × 0.7 = 3.23
The dot product: pair the numbers up, multiply each pair, add.
- Length of the question√(1.44 + 0.25) = 1.3
Square each number, add, take the square root.
- Length of the passage√(5.76 + 0.49) = 2.5
The passage's list is longer, which says nothing about its meaning.
- Divide by both lengths3.23 ÷ (1.3 × 2.5) = 0.99
What's left is direction alone, and 1 would mean exactly the same direction.
Answer0.99, with no word in common.
The model was trained on many pairs of texts that mean the same thing, so it gives "money back" and "refund" nearby directions. Dividing by the lengths matters because length isn't meaning: a passage shouldn't win just by having bigger numbers.
Embedding and cosine similarity
An embedding is the list of numbers a model gives a text. Cosine similarity scores two embeddings by direction only: 1 for the same direction, 0 for unrelated, −1 for opposite.
cos = (a · b) ÷ (|a| × |b|)
Some models return vectors of length 1 already, and you can scale any vector to length 1 once, when you store it. Then both lengths are 1 and the dot product is the cosine, which saves a division on every comparison.
Your turn
"Set up your new laptop" embeds to (0.7, 2.4). What's its cosine with the question (1.2, 0.5)?
Part 2. Search is finding the nearest
Rank passages by cosine, and say what gets embedded when.
To search, embed every passage once, when you build the index, and store the vectors. When a question arrives, embed just the question, score it against the stored vectors, and return the closest few. The heavy work happens once; each search costs one short embedding.
Example 2.1
Ranking three passages
The index holds the refund passage (2.4, 0.7), "Laptop insurance pays for repairs" (0.8, 1.5) and "Set up your new laptop" (0.7, 2.4). Rank them for the question (1.2, 0.5), whose length is 1.3.
- Refund passage3.23 ÷ (1.3 × 2.5) = 0.99
Worked out in part 1.
- Laptop insurance1.71 ÷ (1.3 × 1.7) = 0.77
Dot product 1.2 × 0.8 + 0.5 × 1.5 = 1.71. Length √(0.64 + 2.25) = 1.7.
- Laptop setup2.04 ÷ (1.3 × 2.5) = 0.63
Your turn from part 1.
- Sort, highest first0.99, 0.77, 0.63
The top result shares no words with the question.
AnswerRefund, then insurance, then setup.
Dense retrieval
Search by comparing embeddings. "Dense" because every number in an embedding carries something. BM25's view of a passage is sparse: a count for every word in the vocabulary, almost all of them 0.
One rule is strict: questions and passages must go through the same model. Each model learns its own map of meaning, so a vector from one means nothing in another's space, even when the lists have the same length. Change models and you re-embed every passage.
Dense retrieval has the opposite blind spot to BM25. To an embedding model, the codes E-4012 and E-4021 look almost alike: short strings in the same format, in the same kind of sentence. Exact codes, part numbers and rare names are where it slips.
Your turn
A new question, "is a cracked laptop screen covered?", embeds to (0.6, 0.8), which has length 1. Which of the three passages comes first?
Part 3. Searching only the nearest buckets
Work out what bucket search saves, and what it can miss.
Comparing the question with every passage is fine for three. Now take a million passages of 768 numbers each: one search means 768 million multiply-adds. The way out is to give up a little accuracy and check only the passages likely to be close.
Example 3.1
A million passages in buckets
Ahead of time, a million passages are grouped into 1,000 buckets of similar ones, each with a centre point. A search compares the question with every centre, then checks every passage in the 10 nearest buckets. How many comparisons does one search make?
- Passages per bucket1,000,000 ÷ 1,000 = 1,000
About a thousand each, if the buckets are even.
- Find the nearest buckets1,000
One comparison with each centre, keeping the closest 10.
- Search inside them10 × 1,000 = 10,000
Every passage in the 10 chosen buckets.
- Add up1,000 + 10,000 = 11,000
About 1.1% of the million comparisons a full scan makes.
Answer11,000 comparisons, not 1,000,000.
Approximate nearest neighbour search
Finding passages that are very probably the closest without checking them all, ANN for short. The bucket method is called IVF. In the Faiss library, nlist is the number of buckets and nprobe the number searched.
comparisons ≈ nlist + nprobe × N ÷ nlist
The price is misses. A passage near the edge of its bucket can be closer to the question than everything in the buckets you searched. Searching more buckets misses less and costs more. Faiss searches one bucket unless you ask for more, so measure what you lose before you trust it.
Many vector databases use a graph index called HNSW instead, covered in "Indexes at scale with HNSW". It makes the same trade: a much faster search for an occasional miss.
Your turn
4,000,000 passages sit in 2,000 buckets. A search checks every centre, then every passage in the 20 nearest buckets. How many comparisons is that?
Where you'll meet it
A common design question: why not use embeddings for everything? Say what each retriever misses: dense search blurs exact codes and rare names, and BM25 misses paraphrases. Then say what the index trades: approximate search answers fast but can skip a true neighbour, so you measure how often.
The small open model all-MiniLM-L6-v2 gives 384 numbers per text and runs on a laptop. Some models want a label on each input: the E5 family expects "query: " before questions and "passage: " before passages, and its authors warn of worse results without them. Test a model on your own questions before you pick it.
import math
def cosine(a, b):
dot = sum(x * y for x, y in zip(a, b))
return dot / (math.hypot(*a) * math.hypot(*b))
index = { # embedded once, when the index is built
"refund": (2.4, 0.7),
"laptop insurance": (0.8, 1.5),
"laptop setup": (0.7, 2.4),
}
for question in [(1.2, 0.5), (0.6, 0.8)]: # embedded as each arrives
ranked = sorted(index, key=lambda p: -cosine(question, index[p]))
print([(p, round(cosine(question, index[p]), 2)) for p in ranked])
# [('refund', 0.99), ('laptop insurance', 0.77), ('laptop setup', 0.63)]
# [('laptop insurance', 0.99), ('laptop setup', 0.94), ('refund', 0.8)]math.hypot gives a vector's length. With real embeddings you'd use NumPy or a vector index, but the ranking is the same: sort by cosine.
Check yourself
Four questions from across the lesson. Try each one before you look back.
1
You embed the passages with one embedding model and the question with a different one. What happens?
2
Passage A embeds to (1, 2) and passage B to (2, 4). Which has the higher cosine similarity with a question?
3
Which query is dense retrieval most likely to get wrong?
4
What do you give up with the bucket search?
What to remember
1Meaning as a direction
Cosine similarity is the dot product over both lengths: (1.2, 0.5) and (2.4, 0.7) give 3.23 ÷ (1.3 × 2.5) = 0.99, with no word in common.
2Search is finding the nearest
Embed every passage once, at indexing, and only the question at search time, with the same model on both sides.
2Search is finding the nearest
Dense retrieval finds paraphrases that BM25 misses, and blurs exact codes like E-4012 and E-4021 that BM25 gets right.
3Searching only the nearest buckets
Bucket search (IVF) checks 1,000 centres and 10 buckets, 11,000 comparisons instead of a million, at the price of an occasional miss.
Recap and going further
The lesson on one page
- Text in, a list of numbers out
- Similar meaning, similar direction
- Dot product ÷ both lengths
- 1: same direction; length plays no part
- Passages embedded once, at indexing
- The question embedded when it arrives
- One model for both sides
- Compare with every centre, search the nearest buckets
- 1,000 + 10,000 comparisons, not a million
- nprobe trades speed for misses
- Exact codes and rare names blur
- Which is why BM25 stays
Try this
Build a small semantic search
Install sentence-transformers and load all-MiniLM-L6-v2. Embed the same 50 to 200 paragraphs you used for BM25 (or any page split into paragraphs), embed ten real questions, and rank the paragraphs by cosine similarity. Run the same questions through BM25 and put the two top-3 lists side by side.
You need: Python with sentence-transformers. It runs on a laptop CPU.
Look for
- Questions dense search gets right that BM25 missed, and the other way round.
- What happens to a question that contains an exact code or product number.
- How close together the top five scores are.
- What breaks if you embed the questions with a different model from the paragraphs.
The words, in plain language
- Embedding and cosine similarity
- An embedding is the list of numbers a model gives a text. Cosine similarity scores two embeddings by direction only: 1 for the same direction, 0 for unrelated, −1 for opposite.
- Dense retrieval
- Search by comparing embeddings. "Dense" because every number in an embedding carries something. BM25's view of a passage is sparse: a count for every word in the vocabulary, almost all of them 0.
- Approximate nearest neighbour search
- Finding passages that are very probably the closest without checking them all, ANN for short. The bucket method is called IVF. In the Faiss library, nlist is the number of buckets and nprobe the number searched.
Read next
- Sentence-BERT: Sentence Embeddings using Siamese BERT-NetworksReimers and Gurevych, 2019The paper behind most open sentence-embedding models used for search.
- Dense Passage Retrieval for Open-Domain Question AnsweringKarpukhin and colleagues, 2020Question and passage encoders trained for search, which beat a Lucene BM25 system by 9 to 19 points on top-20 accuracy.
- Semantic SearchSentence Transformers documentationWorking code to embed a collection and search it, with advice on choosing a model.
- Faster searchFaiss wiki, on GitHubThe bucket index, IndexIVFFlat, with its nlist and nprobe settings, in a few lines of Python.
Search and Retrieval overviewSpotted a mistake? Tell us.