Multiple testing in experiments

Evaluation, lesson 12

Try enough variants and one will win by luck.

"Is the new version really better": an interval and a paired test for comparing two versions. This lesson is about what happens when you compare many.

Part 1. Twenty variants, one winner

It's Friday evening. Your current prompt gets 70% of a 500-question dev set right, and you have 20 ideas for doing better: reworded instructions, other examples, a new output format. A script runs all 20 overnight. It compares each with the current prompt on the same 500 questions and flags any variant whose gain passes a significance test at p < 0.05 (a one-sided test: is the variant better?).

In the morning one variant is flagged: +3.0 points, p = 0.04. Before celebrating, suppose for a moment that none of the 20 ideas does anything at all.

Your turn

If all 20 variants are truly no better than the current prompt, how many would you expect the script to flag?

Each test has a 5% chance of a false alarm when there's nothing to find. Twenty tests give 20 × 0.05 = 1 false win on average. So one flagged variant in the morning is exactly what luck alone produces.

Show the answer

C. Each test has a 5% chance of a false alarm when there's nothing to find. Twenty tests give 20 × 0.05 = 1 false win on average. So one flagged variant in the morning is exactly what luck alone produces.

Example 1.1

At least one false win in 20 tries

one test, nothing real→stays quiet with chance 0.95
A test at the 5% level stays quiet 95% of the time when the variant does nothing.
all 20 quiet=0.95²⁰=0.36
Independent chances multiply. Read 0.95²⁰ as 0.95 multiplied by itself 20 times.
at least one false win=1 − 0.36=0.64
About two nights in three, at least one useless variant gets flagged.
expected false wins=20 × 0.05=1
with 100 variants=5
One a night on average, five if you try a hundred. (The variants share a dev set, so the tests aren't fully independent, but the picture is the same.)

Multiple testing

Running many tests and paying attention to the ones that pass. The more you run, the more false winners you get. The chance of at least one across the batch is the family-wise error rate: 1 − (1 − α)ᵐ for m independent tests at level α (read "alpha", one test's false-alarm rate).

Try it. Every dot below is a variant that does nothing. Run a few nights at 20 variants, then 100. Then switch the bar to Bonferroni (the next section explains it), and retest the flagged ones on fresh questions.

A night of variants that do nothingLive

Part 2. Winners shrink

There's a second problem, and it bites even when some ideas are real. How big is the luck? Say each variant disagrees with the current prompt on 15% of the questions. Then a variant that does nothing still lands a few points either side of zero: one wobble is √(0.15 ÷ 500) ≈ 0.017, or 1.7 points. The best of 20 such draws lands, on average, 1.87 wobbles above zero. That's +3.2 points, and the flagged variant showed +3.0.

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come

Winners shrink
Raise the bar, and keep a set you never touch
The overnight agent is a multiple-testing machine

See Pro plans I have Pro: sign in

Next lessonTreating prompts like codeVersion prompts and run regression tests before every change ships. Start lesson 13

Evaluation overviewSpotted a mistake? Tell us.

12 · Multiple testing in experiments