Measuring search with recall, MRR and NDCG

Search and Retrieval, lesson 8

Score a ranked list, and pick the metric that matches the job.

"Reranking": recall for the first stage, precision for the second. This lesson turns them, and two more measures, into numbers you can track.

Part 1. One ranked list, scored

To measure a retriever you need questions where you already know the right passages. Take one: "How long do enterprise customers have to get a refund?" Someone reads the handbook and marks two passages. Enterprise refunds answers it. Annual refunds is related, and useful for contrast.

The retriever returns five passages, in this order: 1 kitchen rules, 2 enterprise refunds, 3 laptops, 4 travel, 5 annual refunds.

Relevance labels

The passages a person has marked as right for a question, sometimes with grades (here 2 for "answers it", 1 for "related"). A set of questions with labels is a test set, and every metric in this lesson compares a ranked list against it.

Your turn

Of the two marked passages, how many are in the top 3?

Enterprise refunds is at rank 2, inside the top 3. Annual refunds is at rank 5, outside it. One of two found: 0.5. Look at the top 5 instead and both are in, so recall at 5 is 1.0.

Show the answer

A. Enterprise refunds is at rank 2, inside the top 3. Annual refunds is at rank 5, outside it. One of two found: 0.5. Look at the top 5 instead and both are in, so recall at 5 is 1.0.

Part 2. Recall@k: did it make it in?

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come

Recall@k: did it make it in?
MRR: how high is the first hit?
NDCG: order and grades together

See Pro plans I have Pro: sign in

Next lessonAnswering from documentsRetrieve passages, then have the model answer from them. Start lesson 9

Search and Retrieval overviewSpotted a mistake? Tell us.

08 · Measuring search with recall, MRR and NDCG