p-values

Twenty teams test a pill that does nothing at all. Guess how many find it “significant”, then watch their results land.

A pill that does nothing

Before the clip you guessed. The pill does nothing, yet about one study in twenty comes out “significant” at 0.05. In the clip’s run, 1 of the 20 did. That is not a fault in the studies. It is what the 5 % false-alarm level of Significance tests promises: when nothing is going on, 5 % of tests still cross the line.

Look at where the p-values land: spread evenly along the line from 0 to 1, with no pile anywhere. That is also no accident, as the section on even spreads below shows.

The p-value

A test with a fixed threshold says only “above” or “below”. A p-value says how far into the tail the result fell:

It is a chance computed as if $H_0$ were true. I write it $p_\text{val}$, because a bare $p$ is a chance of success on this site. A small $p_\text{val}$ means: if nothing were going on, data this extreme would be rare. Rejecting $H_0$ whenever $p_\text{val}<0.05$ is the same as the 5 % test of the last page.

A coin. A coin said to be fair lands heads 61 times in 100 flips. If it is fair, the count is $\mathrm{Bin}(100,\tfrac12)$, and the chance of 61 or more heads is

That is the one-sided $p_\text{val}$. If a result as far below 50 would have been just as surprising, count both tails: $2\times0.0176=0.0352$.

The bell gives nearly the same. With the half-bar step of The central limit theorem, $z_\text{obs}=(60.5-50)/5=2.1$, so the one-sided tail is $1-\Phi(2.1)=0.0179$ and the two-sided $0.0357$.

Shuffling: a p-value you can count

Two page layouts were timed on four people each. The old layout took 10, 11, 12 and 16 seconds, mean 12.25; the new one took 14, 15, 17 and 19 seconds, mean 16.25. The new layout looks 4.0 seconds slower. Could that be chance?

If the layout makes no difference, the labels “old” and “new” are arbitrary: any four of the eight times could have been the “new” ones. So deal the eight times into two groups of four in every possible way, and see how often the difference is as large as 4.0.

There are $\binom84=70$ ways to choose which four times are “new”. Only 3 of them give a difference of 4.0 or more. So

one-sided, and $6/70=0.0857$ counting a difference of 4.0 in either direction. This is a shuffle test, or permutation test: the p-value is a fraction of shuffles you can count. With 4 values per group the smallest p-value it can ever give is $1/70$.

What a p-value is not

p-values when $H_0$ is true

Why did the twenty p-values spread evenly? For a continuous statistic, when $H_0$ is true,

A p-value below 0.05 happens 5 % of the time, below 0.5 half the time, below 0.01 one time in a hundred. The threshold on the p-value is a threshold on the statistic in other units, so the chance of falling below it is exactly its level.

With a real effect, p-values pile up near 0. For the bells 3 apart of Significance tests, a one-sided p-value falls below 0.05 with chance $0.9123$, below 0.01 with chance $0.7497$, and below 0.001 with chance $0.464$.

Many tests

Run 20 independent tests of things that do nothing, each at 0.05. The chance that at least one crosses the line is

More likely than not, something looks significant. With $m$ independent tests it is $1-0.95^m$.

The simplest fix, the Bonferroni correction, tests each of the $m$ at $0.05/m$. For 20 tests that is $0.0025$ each, and, for independent tests, the chance of any false alarm falls to $1-0.9975^{20}=0.0488$, just under 5 %. After the clip, drag the number of studies and watch the lower line: it is the Bonferroni threshold.

How many discoveries are false

A small p-value does not say how likely an effect is to be real. That depends on how many of the ideas tested were real to begin with, the base rate of Bayes’ theorem.

Picture 1000 pills tested, of which 10 % really work. With power 0.8, the tests find 80 of the 100 that work. Of the 900 that do nothing, 5 %, or 45, also pass. So 125 pills come out “significant”, and 45 of them, 36 %, do nothing.

In general, if a fraction $q$ of the ideas tested are real, the share of “discoveries” that are false is

Here $q$ is a local letter for that share.

Worked example

1. The coin. For 61 heads in 100 flips: one-sided $p_\text{val}=0.0176$, two-sided $0.0352$. The test says that a fair coin would rarely do this. It does not say the coin is fair with chance 0.035.

2. The layouts. The shuffle test gives $3/70=0.0429$ one-sided and $6/70=0.0857$ two-sided. Whether the new layout counts as slower at the 5 % level depends on which question was fixed before the data: “slower” or “different”.

3. Twenty studies. Expected significant results: 1. Chance of at least one: $0.6415$. With Bonferroni at $0.0025$ each: $0.0488$.

4. The pills. With 10 % real effects, power 0.8 and $\alpha=0.05$: 36 % of the significant results are false.

Try it yourself. In 5 tests of nothing at 0.05, the chance of at least one false alarm is $1-0.95^5=0.2262$. A coin that lands heads 7 times in 7 flips has one-sided $p_\text{val}=1/128=0.0078$. Under $H_0$, the chance that $p_\text{val}<0.5$ is 0.5. In 100 tests of useless pills, expect 5 significant.

Where you’ll meet this

Reporting experiments. Papers in medicine, psychology and biology report p-values for their main results. Many journals now ask for effect sizes and intervals beside them, because a p-value alone says so little.

A/B tests. Checking a running test every day and stopping at the first $p_\text{val}<0.05$ is many tests in disguise, and it collects false wins.

Genetics. A genome scan tests hundreds of thousands of positions at once, so it uses thresholds far below 0.05, the Bonferroni idea at scale.

Headlines. “A study found a link” often means one test of many crossed 0.05. The base rate of real links among those tested decides how much it means.

Reference card