Significance tests

A test sets a threshold on one number before the data arrive. Slide it, and watch two kinds of mistake trade places.

Two hypotheses and a threshold

A test starts from a question with two possible answers. The null hypothesis, $H_0$, says nothing is going on: the pill does nothing, the machine is set right, there is no tone in the noise. The alternative, $H_1$, says there is an effect.

From the data I compute one number, the test statistic. Here it is $Z$, and it behaves differently under the two answers. If $H_0$ is true, $Z$ follows bell A, the standard bell $\mathcal{N}(0,1)$. If $H_1$ is true, it follows bell B, the same bell shifted to 3. I call that shift $\delta$; here $\delta=3$.

Before looking at the data I fix a threshold $c$. If $Z$ lands above it I reject $H_0$, deciding there is an effect; otherwise I do not. The values above $c$ are the rejection region. This test is one-sided: only large values of $Z$ count against $H_0$.

Two mistakes are possible:

The two shaded areas in the picture are those chances:

with $\Phi$ the distribution function of the standard bell, as in The normal distribution. The false-alarm chance $\alpha$ is the test’s significance level. Here $\alpha$ and $\beta$ are this page’s two error chances; signal processing pages use the same letters for other things.

Moving the threshold

Before the clip you guessed. As the threshold moves right, false alarms fall and misses rise. The two areas trade places: any threshold that cuts one makes the other larger.

Halfway between the bells, at 1.5, the two errors are equal. The usual choice is to fix the false alarms first, at 5 %, with the threshold $z_{0.95}=1.645$: the point of the standard bell with 0.95 of the area below it. With the bells 3 apart, that leaves misses at $0.0877$.

Power, and what raises it

The chance of catching a real effect, $1-\beta$, is the test’s power. At the 5 % threshold with $\delta=3$ it is $0.9123$.

Moving the threshold cannot raise power without raising false alarms. Only pulling the bells further apart helps both. How far apart they sit depends on the data.

Say I test whether a mean exceeds a value $\mu_0$, with known spread $\sigma_X$ and $n$ draws. The statistic is the sample mean in units of its standard error, from Confidence intervals:

If the true mean is $\mu_0+\Delta$, bell B sits at $\delta=\Delta\sqrt n/\sigma_X$. A bigger effect $\Delta$ pulls it out; so does more data, but only as $\sqrt n$.

Four times the data doubles the separation. For an effect of half a standard deviation at the 5 % level:

To reach a chosen power, solve for $n$. With $z_{1-\alpha}$ and $z_{1-\beta}$ the matching points of the standard bell,

For power 0.80 at 5 %, $z_{0.95}=1.6449$ and $z_{0.80}=0.8416$. An effect of half a standard deviation needs $\big((1.6449+0.8416)/0.5\big)^2=24.73$, so 25 draws; an effect of 0.2 standard deviations needs 154.6, so 155.

Worked example

1. A bottling line. The label says 500 g. The filling machine’s spread is known, $\sigma_X=4$ g. Each hour, 16 bottles are weighed, so the standard error is $4/\sqrt{16}=1$ g. The machine is flagged if the mean is too low, with 5 % false alarms: when $\bar x<500-1.645=498.355$ g.

If the machine has drifted to 498 g, two standard errors low, the flag fires with chance $\Phi(0.35515)=0.6388$. At 499 g it fires with chance $0.2595$. To catch 498 g with power 0.80 takes 25 bottles: the standard error falls to 0.8 and the power to $0.8038$.

An hour’s 16 bottles average 497.6 g. That is below the line, so the machine is flagged.

2. Finding a tone. A sine of amplitude 0.5 that completes whole cycles in 72 samples is buried in noise of variance 1. Its energy is $72\times0.25/2=9$. A matched filter that knows the tone’s frequency and phase turns the record into a statistic that follows bell A with noise alone and bell B, shifted by $\sqrt9=3$, with the tone: the bells of the picture.

A threshold of 1.645 raises a false alarm on 5 % of noise-only records and finds the tone 91.2 % of the time. A threshold of 3 cuts false alarms to 1 record in 741 and finds the tone only half the time. With an unknown phase the bells change shape; the matched-filter page deals with that case.

Try it yourself. With the bells 3 apart, the false-alarm chance at threshold 2 is $0.0228$. Doubling $n$ moves bell B to $3\sqrt2=4.243$, and the power at 5 % rises to $0.9953$. Raising the threshold lowers the power.

Where you’ll meet this

Clinical trials. A trial is sized before it starts: the false-alarm level is fixed, usually at 5 %, and the number of patients is chosen for a power of 80 or 90 % to detect the smallest effect that would matter.

Quality control. Production lines are checked by samples against limits, as with the bottles. The limits are set for a tolerable rate of false stops.

Detectors. Radar, sonar and smoke alarms decide “target” or “no target” by a threshold set for a false-alarm rate, then live with the misses that follow.

Experiments online. A/B tests compare two versions of a page; running one without enough visitors gives a test with little power, whose “no difference” means very little.

The next page, p-values, reports how far into bell A’s tail the observed statistic fell, instead of only “above or below the line”.

Reference card