Search and Retrieval, lesson 3
Combining keyword and vector search
Fuse two ranked lists into one that beats both.
BM25 misses questions asked in other words, and dense search blurs exact codes. Their misses fall in different places, so the natural move is to run both and merge what they find. You need BM25 and cosine similarity from the last two lessons, and recall from "The confusion matrix".
By the end, you can:
Part 1. Scores on different scales
Explain why adding a BM25 score to a cosine score fails.
Take one question and run both retrievers. Each returns passages with scores. The obvious merge is to add each passage's two scores and sort. Look at what the scores are first.
Example 1.1
Adding the two scores
For one question, BM25 scores passages A, B and C 8.2, 6.1 and 5.9. Their cosine scores are 0.61, 0.79 and 0.82, so dense search ranks them the other way round. You add each passage's two scores. What order comes out?
- Add for A8.2 + 0.61 = 8.81
BM25's favourite, dense search's least favourite.
- Add for B6.1 + 0.79 = 6.89
Second in both lists.
- Add for C5.9 + 0.82 = 6.72
Dense search's favourite.
- SortA, B, C
Exactly BM25's order. Dense search changed nothing.
AnswerA, B, C: as if dense search never ran.
BM25 scores have no fixed top: they grow with IDF and with the number of search words, so 8 or 20 is normal. Cosine scores stay between −1 and 1, and a real model's scores often bunch closer still. A gap of 2 in BM25 swamps any gap the cosine can make.
Hybrid search
Running a keyword retriever, like BM25, and a dense retriever on the same question, then merging their ranked lists into one.
You could rescale each list so its top score is 1 and its bottom 0, then add. The vector database Weaviate does this by default. But a rescaled score depends on what else is in the list: the best of three poor passages still gets 1. The next part merges by rank instead.
Your turn
BM25 gives passages P and Q 12.4 and 9.0. Dense search gives them cosines 0.35 and 0.90, so it strongly prefers Q. You add the scores. Which comes first?
Part 2. Merging by rank
Fuse two ranked lists by hand with reciprocal rank fusion.
One thing means the same in both lists: position. First place is first place, whatever the scores. So give each passage a small vote from every list it's in, shrinking as its rank goes down, and add the votes up.
Example 2.1
Fusing two lists
For "card declined E-4012", BM25 returns: E-4012 error, Card billing dates, Update your card, E-4021 error, Expense claims. Dense search returns: Payment failed, E-4012 error, Card billing dates, Update your card, Contact support. Each list votes 1 ÷ (60 + rank). What's the fused order?
- Votes for ranks 1 and 21 ÷ 61 = 0.0164, 1 ÷ 62 = 0.0161
Ranks 3, 4 and 5 get 0.0159, 0.0156 and 0.0154. The 60 keeps votes close, so fifth place counts nearly as much as first.
- E-4012 error, in both0.0164 + 0.0161 = 0.0325
First in BM25, second in dense.
- Card billing dates, in both0.0161 + 0.0159 = 0.0320
Second and third.
- Update your card, in both0.0159 + 0.0156 = 0.0315
Third and fourth.
- Passages in one list0.0164, 0.0156, 0.0154
Payment failed (dense's first), then E-4021 error (BM25's fourth), then Expense claims and Contact support, tied.
AnswerE-4012 error, Card billing dates, Update your card, Payment failed, E-4021 error.
Agreement wins. A passage found by both lists gets two votes, about 0.03, and beats any passage found by one, 0.0164 at best. Within each group, better ranks win. No score from either retriever is used, so their scales can't get in the way.
Reciprocal rank fusion (RRF)
Score each passage by adding 1 ÷ (k + rank) over every list it appears in, then sort. k = 60 comes from the 2009 paper that introduced RRF, whose authors found it near the best and the exact choice not critical.
1 ÷ (k + rank₁) + 1 ÷ (k + rank₂)
k sets the balance. A large k makes the votes nearly equal, so agreement decides. A small k makes first place worth far more, so each list's top pick decides. And a list with no results casts no votes: if BM25 finds nothing, the fused list is the dense list.
Your turn
Run the same two lists with k = 1 instead of 60. A rank-1 vote is now 1/2 and a rank-5 vote is 1/6. What changes?
Part 3. Measuring whether fusion helps
Work out recall@k on a small set of test questions, and compare retrievers by it.
Fusion should miss less than either list alone. Don't take that on trust: measure it. Write down real questions, mark the passage that answers each, and count how often each retriever puts that passage near the top.
Example 3.1
Five test questions
Five questions, each with one right passage. The model is shown the top 3. Where did each retriever put the right passage (– means not found)? BM25: 1, 1, –, 4, 2. Dense: 2, –, 1, 2, 5. Fused: 1, 4, 3, 3, 3. Which finds the most?
- BM25's top 33 ÷ 5 = 0.6
Questions 1, 2 and 5. It missed the train-ticket question and put the refund passage 4th.
- Dense top 33 ÷ 5 = 0.6
Questions 1, 3 and 4. It never found the E-4021 page, and put the repairs page 5th.
- Fused top 34 ÷ 5 = 0.8
Every question but 2. The E-4021 page, BM25's first pick, fell to 4th behind three passages both lists found.
AnswerFused 0.8, each list alone 0.6.
Recall@k
The share of the right passages that appear in the top k results, averaged over the test questions. With one right passage per question, it's the share of questions whose passage made the top k.
recall@3 = 4 ÷ 5 = 0.8
Read the misses as well as the average. Question 2 shows what RRF costs: a passage only one list found, even in first place, can sink below passages both lists found lower down. If exact codes matter to your users, look at those questions before you ship.
recall@k only asks whether the passage made the top k, not where. Rank 3 counts the same as rank 1. "Measuring search with recall, MRR and NDCG" adds scores that care about position.
Your turn
A retriever is tested on 8 questions, one right passage each. It put them at ranks 1, 3, 2, –, 7, 1, 4 and 2 (– means not found). What are its recall@3 and recall@5?
Where you'll meet it
Asked how to combine keyword and vector search, say why raw scores don't add (different scales), describe RRF with k = 60, and say how you'd check it: recall@k on labelled questions, for each retriever alone and fused. Name the cost too: one list's confident single find can sink.
Search engines ship fusion. Elasticsearch has an RRF retriever whose rank constant defaults to 60, and OpenSearch added RRF in version 2.19, with optional weights per list. Weaviate defaults to rescaled scores instead. Give one list more weight only when your own recall numbers say it helps.
def rrf(rankings, k=60):
scores = {}
for ranking in rankings:
for rank, doc in enumerate(ranking, start=1):
scores[doc] = scores.get(doc, 0) + 1 / (k + rank)
return sorted(scores, key=lambda d: -scores[d])
bm25 = ["E-4012 error", "Card billing dates", "Update your card",
"E-4021 error", "Expense claims"]
dense = ["Payment failed", "E-4012 error", "Card billing dates",
"Update your card", "Contact support"]
print(rrf([bm25, dense])[:3])
print(rrf([bm25, dense], k=1)[2])
def recall_at(k, ranks): # rank of each right passage, None if missed
return sum(r is not None and r <= k for r in ranks) / len(ranks)
print(recall_at(3, [1, 1, None, 4, 2]), recall_at(3, [2, None, 1, 2, 5]),
recall_at(3, [1, 4, 3, 3, 3]))
# ['E-4012 error', 'Card billing dates', 'Update your card']
# Payment failed
# 0.6 0.6 0.8With k = 1, "Payment failed" moves up to third, past "Update your card". The recall_at function needs only the rank of each right passage, so the same ten lines score any retriever you try.
Check yourself
Three questions from across the lesson. Try each one before you look back.
1
Your users ask both kinds of question, and you can't tell in advance which kind is coming. What's the sensible setup?
2
Someone asks "can I claim a train ticket?". BM25 returns nothing at all, and dense search returns five passages. What does reciprocal rank fusion return?
3
On 10 test questions, fused search has recall@5 of 0.8 and BM25 alone 0.7. What does that gap come down to?
What to remember
1Scores on different scales
BM25 scores have no fixed top and cosine stays between −1 and 1, so a raw sum follows BM25: 8.81, 6.89 and 6.72 kept its order exactly.
2Merging by rank
Reciprocal rank fusion adds 1 ÷ (60 + rank) from each list, so a passage both lists found (about 0.032) beats any found by one (0.0164 at most).
2Merging by rank
An empty list casts no votes, and a small k lets each list's top pick decide.
3Measuring whether fusion helps
recall@k is the share of right passages in the top k: fusion found 4 of 5 in its top 3 (0.8), against 3 of 5 (0.6) for each list alone.
Recap and going further
The lesson on one page
- BM25 misses paraphrases
- Dense search blurs exact codes
- BM25 has no fixed top; cosine stays within −1 to 1
- Their sum is BM25 in disguise
- Each list votes 1 ÷ (k + rank)
- k = 60: agreement wins
- Passages both lists found rise to the top
- An empty list adds no votes
- recall@k on labelled questions
- Read the misses as well as the average
Try this
Fuse your own two searches
Take the BM25 and embedding rankings from the last two lessons' tries (or any two ways of ranking the same paragraphs) for ten questions. Mark the paragraph that answers each. Write reciprocal rank fusion in about ten lines, then work out recall@3 for BM25, for embeddings and for the fused list. Rerun with k = 1, and with one list's votes doubled.
You need: Python, and two rankings of the same paragraphs.
Look for
- Questions where the fused top 3 beats both single lists.
- Any question where fusion pushes the right paragraph down.
- How the order shifts between k = 1 and k = 60.
- Whether doubling one list's votes helps across your questions or only on a few.
The words, in plain language
- Hybrid search
- Running a keyword retriever, like BM25, and a dense retriever on the same question, then merging their ranked lists into one.
- Reciprocal rank fusion (RRF)
- Score each passage by adding 1 ÷ (k + rank) over every list it appears in, then sort. k = 60 comes from the 2009 paper that introduced RRF, whose authors found it near the best and the exact choice not critical.
- Recall@k
- The share of the right passages that appear in the top k results, averaged over the test questions. With one right passage per question, it's the share of questions whose passage made the top k.
Read next
- Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning MethodsCormack, Clarke and Büttcher, 2009The two-page paper that introduced reciprocal rank fusion and its k = 60.
- Hybrid Search ExplainedWeaviate blogHow one vector database merges keyword and vector results, by rank or by rescaled score.
- Reciprocal rank fusionElasticsearch documentationRRF as a built-in retriever over a BM25 query and a vector search, with its rank_constant and window size.
Search and Retrieval overviewSpotted a mistake? Tell us.