insideout Get Pro

04 · Multiple testing

Pro

Statistics and Regression, lesson 4

Multiple testing

Try twenty ideas and one will look significant by luck.

A strategy that passes a 5% test looks like a find, but if it was the best of twenty tries, luck alone may explain it. This lesson counts how often luck produces a winner, and how to raise the bar. You need p-values and the 5% line from the last lesson.

By the end, you can:

  1. Work out the chance that at least one of many useless tests passes.
  2. Work out how often the best of many useless signals looks as good as yours.
  3. Correct for many tests with the Bonferroni bar.
  4. Keep the share of false finds low with the Benjamini-Hochberg rule.

Part 1. Twenty tries, one false winner

Work out the chance that at least one of many useless tests passes.

The last lesson ended on a number: twenty ideas with no edge, each tested at 5%, give at least one "significant" result 64% of the time. Here is where it comes from. Picture twenty strategies built from coin flips, so none has an edge, each tested on a year of daily returns.

Example 1.1

Twenty strategies with no edge

You test 20 strategies, none with an edge, each at the 5% level. The tests are independent. How likely is at least one false winner? How many do you expect?

  1. One test stays quiet1 − 0.05 = 0.95

    A useless strategy passes a 5% test 5% of the time, so it stays quiet 95% of the time.

  2. All twenty stay quiet0.95²⁰ = 0.36

    Independent chances multiply: 0.95 times itself twenty times.

  3. At least one passes1 − 0.36 = 0.64

    Every outcome other than "all quiet" has a winner in it.

  4. Count the expected winners20 × 0.05 = 1

    Each test adds 0.05 false winners on average.

Answer64%, about two times in three, and one false winner expected.

Each test on its own is fair. The trouble starts when you look at twenty results and point to the one that passed. You've let twenty chances compete, and the winner isn't special. It's the luckiest of twenty.

25%50%75%100%0020406080100tests at 5%, none with an edge1 test: 5%20 tests: 64%50 tests: 92%100 tests: 99.4%1 − 0.95ᵐat least one false winner
Figure 1The chance that at least one of m useless tests passes at 5%. It's 64% at 20 tests, 92% at 50 and over 99% at 100.

Family-wise error rate

Running many tests and paying attention to the ones that pass is multiple testing. The chance of at least one false positive across the whole family of m tests is the family-wise error rate. At level α, read "alpha", for independent tests:

1 − (1 − α)ᵐ = 1 − 0.95²⁰ = 0.64

Related tests, such as twenty versions of one idea, tend to pass or fail together, so the chance of at least one false winner comes out lower than 1 − 0.95ᵐ. The expected number, m × α, doesn't need independence. It stays one false winner per twenty tests either way.

Your turn

You test 50 signals, none with an edge, each at the 5% level, independently. What's the chance at least one passes?

All 50 stay quiet with chance 0.95⁵⁰ = 0.077, so at least one passes with chance 1 − 0.077 = 0.92. 5% is one test's chance, and 64% is the answer for 20. 50 × 5% = 2.5 is the expected number of false winners, not a chance.

Part 2. How good the luckiest looks

Work out how often the best of many useless signals looks as good as yours.

Keep going with Pro

You've done the first part. Pro opens the rest of this lesson, and every other one.

Still to come
  1. How good the luckiest looks
  2. Raising the bar
  3. A gentler bar for screening
See Pro plansI have Pro: sign in
Next lessonCorrelationMeasuring how two things move together, and how it misleads. Start lesson 5

Statistics and Regression overviewSpotted a mistake? Tell us.