Ablations

Evaluation, lesson 10

Remove one part at a time to learn what each part is worth.

"Baselines": a score only means something next to a simpler system run on the same test set. An ablation turns that comparison inward, on the parts of your own system.

Part 1. Take one piece out

A support bot has four parts besides the model: it retrieves help articles, a reranker sorts them, a few examples in the prompt show the answer style, and a self-check step asks the model to re-read its answer against the articles. All together, it answers 84% of 500 test questions correctly.

Which parts earn their place? Turn one off, rerun the same 500 questions, turn it back on, and repeat for the next part.

Example 1.1

Removing one part at a time

full system→84%
The reference point. Every other row is compared with this one.
no retrieval→51%(−33)
no reranker→82%(−2)
Without articles the bot is guessing about your product, so it collapses. The reranker's effect is small.
no examples→83%(−1)
no self-check→79%(−5)
The examples barely matter for correctness. The self-check catches a fair number of mistakes.

Ablation

Removing (or switching off) one part of a system and measuring how much the score changes. The set of runs is an ablation study. The word comes from surgery, where it means removing tissue.

Your turn

The reranker adds cost and some delay to every request. Based only on those numbers, what's the most sensible next step?

Two points on 500 questions might be real or might be chance, and the decision depends on which. Keeping everything "because it helps a bit" is how systems pile up parts nobody can justify. Retrieval is obviously the part you can't lose.

Show the answer

B. Two points on 500 questions might be real or might be chance, and the decision depends on which. Keeping everything "because it helps a bit" is how systems pile up parts nobody can justify. Retrieval is obviously the part you can't lose.

Part 2. Is the drop bigger than the wobble?

Each ablation is a paired comparison: same questions, two versions. Only the questions where the two versions disagree carry information, so for each part, count the questions only the full system got right and the questions only the cut-down one got right. The code turns those counts into an interval: a range the true drop probably lies in. The next lesson, "Is the new version really better", builds that interval from scratch.

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come

Is the drop bigger than the wobble?
Parts that cover for each other
Ablating a prompt

See Pro plans I have Pro: sign in

Next lessonIs the new version really betterConfidence intervals and paired tests for comparing two systems. Start lesson 11

Evaluation overviewSpotted a mistake? Tell us.

10 · Ablations