Checking answers against sources

Search and Retrieval, lesson 11

Test whether each answer is supported by what was retrieved.

"Measuring search with recall, MRR and NDCG" and "Citing sources": scoring the ranked list, and tying claims to passages. This lesson scores the answer itself, leaning on Evaluation's "Models as graders" without repeating it.

Part 1. Right passages, wrong answer

The retriever does its job. The prompt contains "Enterprise plans get a refund within 60 days. The refund is pro-rated by unused months." The model replies: "Enterprise customers have 90 days and get a full refund."

Recall@5 is 1.0. Every retrieval metric says this was a success. The customer still got the wrong answer.

Your turn

What do retrieval metrics miss here?

Retrieval metrics stop at the ranked list. They can't see that the model changed 60 to 90 and dropped the pro-rating. Scoring a RAG system needs a second set of checks on the answer itself.

Show the answer

C. Retrieval metrics stop at the ranked list. They can't see that the model changed 60 to 90 and dropped the pro-rating. Scoring a RAG system needs a second set of checks on the answer itself.

Faithfulness

Whether every claim in the answer is backed by the retrieved passages. Also called groundedness. An answer can be true in the world and still unfaithful, if nothing retrieved said it.

Part 2. Checking one claim at a time

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come

Checking one claim at a time
Three questions about one answer
A test set that catches regressions

See Pro plans I have Pro: sign in

Next lessonWhy grounded answers still go wrongMissed passages, conflicting sources, and claims the text doesn't support. Start lesson 12

Search and Retrieval overviewSpotted a mistake? Tell us.

11 · Checking answers against sources