Statistics and Regression, lesson 3
Tests and p-values
How surprising your data would be if nothing were going on.
A strategy made money. A coin landed heads 8 times in 10. A test can't tell you whether that was skill or luck, but it answers a narrower question: if nothing were going on, how often would luck alone do this well? You need standard errors and the counting from Probability.
By the end, you can:
Part 1. Assume nothing is going on
Compute a p-value by assuming the dull explanation and counting.
You can't prove a coin is biased. What you can do is assume the dull explanation, a fair coin, and ask how often it would give a result at least as lopsided as yours. If the answer is "rarely", the dull explanation starts to look bad.
Example 1.1
8 heads in 10 flips
A coin lands heads 8 times in 10 flips. If it were fair, how often would 10 flips be at least this lopsided?
- Assume a fair coin2¹⁰ = 1,024 sequences
Every sequence of heads and tails is equally likely.
- Count results at least this extreme8, 9 or 10 heads: 45 + 10 + 1 = 56
C(10, 8) + C(10, 9) + C(10, 10), from Counting outcomes.
- Count the other side2, 1 or 0 heads: another 56
A coin that favours tails is just as suspicious, so 2 heads is as extreme as 8.
- Divide112 / 1,024 = 0.11
About 1 time in 9.
Answerp ≈ 0.11: unusual, but a fair coin does this about 1 time in 9.
Null hypothesis and p-value
The dull explanation you test against is the null hypothesis: here, "the coin is fair". The p-value is the chance, if the null hypothesis were true, of a result at least as extreme as yours. A small p means surprising data.
A p-value is worked out by assuming the null hypothesis is true, so it can't also be the chance that the null hypothesis is true. Getting that would need Bayes' rule and a belief about the coin from before you flipped it.
Counting both tails makes this a two-sided test. If you'd decided in advance that only heads mattered, you'd count one side and get p = 56 / 1,024 = 0.055. Decide which before you look at the data.
Your turn
A coin lands heads 9 times in 10 flips. What's the two-sided p-value, if the coin is fair?
Part 2. Counting in standard errors
Turn a result into a z-score and read its p-value off the bell curve.
With 100 flips, counting sequences gets slow. The bell curve gives a shortcut: measure how far your result is from what the null hypothesis expects, in standard errors.
Example 2.1
60 heads in 100 flips
A coin lands heads 60 times in 100 flips. How surprising is that for a fair coin?
- What a fair coin expects100 × 0.5 = 50 heads
The null hypothesis says half.
- Its standard error√(100 × 0.5 × 0.5) = 5
One flip has variance 0.5 × 0.5 = 0.25, so 100 flips have variance 25.
- Measure the distancez = (60 − 50) / 5 = 2.0
60 heads is 2 standard errors above what a fair coin expects.
- Read the bell curveP(beyond ±2.0) ≈ 0.046
About 4.6% of a bell curve lies more than 2 standard errors from its middle, counting both sides.
Answerz = 2.0, and p ≈ 0.046.
This is the confidence interval from the last lesson, turned around. There you went 1.96 standard errors out from the estimate. Here you ask how many standard errors the result sits from the null hypothesis. Beyond 1.96 either way, p is below 0.05.
Z-score
How far a result sits from what the null hypothesis expects, measured in standard errors, is its z-score. Beyond ±1.96, p is below 0.05. Beyond ±2.576, it's below 0.01.
z = (result − expected) / standard error
The shortcut is rough near the line. Counting exactly, 60 heads in 100 gives p = 0.057, not 0.046, because heads come in whole numbers and the bell curve is smooth. When a verdict hangs on the second decimal, count exactly.
Your turn
A coin lands heads 64 times in 100 flips. What's the z-score, and is p below 0.05?
Part 3. The 5% line and its two errors
Set a significance level, and count the false alarms and misses it allows.
A p-value is a number, and a decision needs a line. You agree on the line before you look, often 5%. A test of 100 flips at 5% flags a coin when z is beyond ±1.96. Any line makes two kinds of mistake.
Example 3.1
What a 100-flip test gets wrong
A 5% test of 100 flips flags 40 heads or fewer, or 60 or more. How often does it flag a fair coin? How often does it miss a coin that lands heads 60% of the time?
- Find the line in heads50 ± 1.96 × 5 = 40.2 to 59.8
Whole numbers outside this range get flagged.
- A fair coin crosses itP(40 or fewer, or 60 or more) = 0.057
Counted exactly. A little above 5%, because heads come in whole numbers.
- Spread of the biased coinexpects 60, SE = √(100 × 0.6 × 0.4) = 4.9
Its results centre on 60, right on the line.
- It stays inside the lineP(41 to 59 heads) = 0.46
Counted exactly, for a coin that lands heads 60% of the time.
AnswerIt flags a fair coin 5.7% of the time, and misses the biased coin 46% of the time.
Significance level
The line you agree on in advance, often 5%, is the significance level. Below it, you reject the null hypothesis and call the result statistically significant. Rejecting a null hypothesis that's true is a false positive (a type I error). Missing a real effect is a false negative (a type II error).
The two errors pull against each other. Move the line to 1% and fewer fair coins get flagged, but more biased ones slip through. Only more data shrinks both. With 400 flips, the same 5% test misses the 60% coin less than 2% of the time.
When nothing is going on, p-values spread about evenly between 0 and 1, so about 5% land below 0.05 by luck alone. One test at 5% is a fair bet. Run many, and some false alarms are close to certain.
Your turn
You test 200 fair coins, each at the 5% level. About how many look biased?
Part 4. Testing a strategy's average
Test whether a strategy's average return is above zero, straight from its Sharpe ratio.
The same test works on a strategy. The null hypothesis is that its true average daily return is zero. The distance from zero in standard errors is called t, and it comes straight from the Sharpe ratio, the average return divided by its standard deviation.
Example 4.1
A Sharpe ratio of 1 over four years
A strategy has an annual Sharpe ratio of 1.0 over four years of daily returns. Is its average return significantly above zero at the 5% level?
- Write tt = x̄ / (s / √n) = (x̄ / s) × √n
The average over its standard error. x̄ / s is the daily Sharpe ratio.
- Make the Sharpe ratio daily1.0 / √252
Over a year the average grows 252 times and the spread √252 times, so a yearly Sharpe ratio is the daily one times √252.
- Multiply by √n(1.0 / √252) × √(4 × 252) = 1.0 × √4 = 2.0
The 252s cancel.
- Read the p-valuet = 2.0 → p ≈ 0.046
With 1,008 days, t behaves like z. Two-sided, just below 0.05.
Answert = 2.0 and p ≈ 0.046: significant, only just.
The 252s always cancel, which leaves a shortcut worth keeping: t is about the annual Sharpe ratio times the square root of the number of years. A Sharpe ratio of 1 needs almost four years to reach t = 1.96. One of 0.5 needs over fifteen.
T-statistic
An estimate divided by its standard error is its t-statistic. For a strategy's average return, with S the annual Sharpe ratio and T the years of daily data:
t = x̄ / (s / √n) ≈ S × √T
Significant isn't the same as real. Daily returns have fat tails, and their volatility comes in bunches. The formula ignores both. And a strategy picked as the best of many backtests carries its luck into its t. A t of 2 from the winner of twenty tries is weak evidence.
Your turn
A strategy has an annual Sharpe ratio of 0.8 over nine years of daily returns. About what's the t-statistic of its average return?
Where you'll meet it
"What is a p-value?" comes up constantly in quant research and data science interviews, along with "60 heads in 100 flips: is the coin fair?" and the Sharpe-to-t shortcut. The common slip is calling the p-value the chance that the null hypothesis is true. Say instead: the chance of data at least this extreme, if it were true.
In 2016 the American Statistical Association published a statement on p-values (Ronald Wasserstein and Nicole Lazar, The American Statistician). Two of its six principles: p-values do not measure the probability that the studied hypothesis is true, and a p-value does not measure the size of an effect or the importance of a result. So a careful report gives the size of an effect alongside its p-value.
from math import comb, erf, sqrt
def p_exact(n, k):
"""Two-sided: the chance a fair coin lands at least as far from n/2 as k did."""
d = abs(k - n / 2)
return sum(comb(n, j) for j in range(n + 1) if abs(j - n / 2) >= d) / 2 ** n
def p_normal(z):
return 1 - erf(abs(z) / sqrt(2)) # two-sided, from the bell curve
for n, k in [(10, 8), (10, 9), (100, 60), (100, 64)]:
z = (k - n / 2) / (sqrt(n) / 2)
print(f"{k:>3} heads in {n:>3}: z = {z:.2f}, exact p = {p_exact(n, k):.4f}, normal p = {p_normal(z):.4f}")
def chance(n, p, lo, hi): # the chance of lo to hi heads, if each flip lands heads with chance p
return sum(comb(n, j) * p ** j * (1 - p) ** (n - j) for j in range(lo, hi + 1))
print(f"100 flips: flags a fair coin {1 - chance(100, 0.5, 41, 59):.3f}, misses a 60% coin {chance(100, 0.6, 41, 59):.3f}")
# A Sharpe ratio of 1 over 4 years: t is about the Sharpe ratio times the square root of the years
t = 1.0 * sqrt(4)
print(f"Sharpe 1 over 4 years: t = {t:.1f}, p = {p_normal(t):.3f}")
# Output:
# 8 heads in 10: z = 1.90, exact p = 0.1094, normal p = 0.0578
# 9 heads in 10: z = 2.53, exact p = 0.0215, normal p = 0.0114
# 60 heads in 100: z = 2.00, exact p = 0.0569, normal p = 0.0455
# 64 heads in 100: z = 2.80, exact p = 0.0066, normal p = 0.0051
# 100 flips: flags a fair coin 0.057, misses a 60% coin 0.457
# Sharpe 1 over 4 years: t = 2.0, p = 0.046With 10 flips the bell-curve shortcut halves the p-value, 0.058 against an exact 0.109. With 100 it's close. Count exactly when you can.
Check yourself
Four questions from across the lesson. Try each one before you look back.
1
Your test gives p = 0.03. Which statement is right?
2
Strategy A's average return has p = 0.001 over 20 years of data. Strategy B's has p = 0.2 over one year. Which has the bigger average return?
3
You move your significance level from 5% to 1%, with the same data. What happens?
4
A strategy averages 0.05% a day over 900 days, with a daily standard deviation of 1.5%. Is its average significantly above zero at the 5% level?
What to remember
1Assume nothing is going on
A p-value assumes the null hypothesis and asks how often luck gives a result at least as extreme: 8 heads in 10 fair flips is 112 / 1,024 ≈ 0.11.
2Counting in standard errors
A z-score is the distance from what the null hypothesis expects, in standard errors: 60 heads in 100 is (60 − 50) / 5 = 2.0, so p ≈ 0.046.
3The 5% line and its two errors
A 5% line flags about 1 in 20 tests of nothing, and a small test misses real effects: 100 flips miss a 60% coin 46% of the time.
4Testing a strategy's average
A strategy's t is about its annual Sharpe ratio times √years: a Sharpe ratio of 1 over four years gives t = 2.0 and p ≈ 0.046.
Statistics and Regression overviewSpotted a mistake? Tell us.