Confidence intervals

A true value sits still. A hundred samples each throw an interval around their own estimate. Watch how many catch it.

Intervals around a fixed true mean

Before the clip you guessed. About 95 of the 100 intervals contain the true mean; a few miss. The clip’s 100 runs gave 98 hits, two misses: ordinary, because the count of hits wobbles around 95 by about 2.

Look at what moves. The line never moves: the true mean is a fixed number, 10. The intervals move, because each one is built from a different random sample. Some are wider than others, because each sample’s spread is different too.

An interval like this is a confidence interval. Its confidence level, here 95 %, is a property of the recipe: if you used the recipe on sample after sample, it would catch the true value in 95 % of them.

Building the interval

The recipe starts from the sample mean $\bar x$. Over repeated samples of size $n$, the sample mean has mean $\mu_X$ and standard deviation $\sigma_X/\sqrt n$, and by The central limit theorem its distribution is close to a bell. Its standard deviation is the standard error of the mean.

For a bell, 95 % of the area lies within 1.96 standard deviations of the centre. I write that edge $z_{0.975}=1.96$: the point of the standard bell with 0.975 of the area below it, leaving 0.025 in each tail. So, before the sample is drawn,

Turn the inequality around: $\bar X_n$ is within $1.96\,\sigma_X/\sqrt n$ of $\mu_X$ exactly when $\mu_X$ is within that distance of $\bar X_n$. That gives the interval

The probability statement is about $\bar X_n$, a random variable, before the data arrive. Once a sample gives a number $\bar x$, the interval is two fixed numbers, and the true mean is either inside it or not.

For other levels, swap the 1.96 for another edge: $0.6745$ for 50 %, $1.2816$ for 80 %, $1.6449$ for 90 %, $2.5758$ for 99 %. After the clip, drag the level: higher confidence needs wider intervals.

When the spread is unknown: the t interval

In practice $\sigma_X$ is unknown too, so I put the sample standard deviation $s$ in its place. That adds a second source of wobble: $s$ itself varies from sample to sample, and with small samples it is sometimes much too small. To keep 95 % coverage, the edge must be wider than 1.96.

The right edge comes from the t law with $n-1$ degrees of freedom, a bell with heavier tails, worked out for bell-shaped data. I write its 0.975 point $t_{0.975,\,n-1}$:

For $n=25$, $t_{0.975,\,24}=2.0639$; for $n=8$, $t_{0.975,\,7}=2.3646$. As $n$ grows it approaches 1.96. The picture uses this t interval for each sample of 25.

What “95 %” does and does not say

What it does say, in one sentence: the recipe that made this interval catches the true value in 95 % of the samples it is used on. The hundred bars in the picture are that sentence, drawn.

A proportion

For a share, such as the fraction of voters for a party, the estimate is $\hat p$, the share in the sample. By Discrete random variables the count of yes answers is binomial, so $\hat p$ has standard error $\sqrt{p(1-p)/n}$, which is estimated by putting $\hat p$ for $p$:

This plain interval works when the counts of yes and no are both reasonably large. With small counts it can run below 0 or above 1. A better recipe, the Wilson interval, solves for the values of $p$ that sit within 1.96 standard errors of $\hat p$, and always stays between 0 and 1. Software computes it; the worked example quotes it.

How much data

With a known spread, an interval for a mean has half-width $z_{0.975}\,\sigma_X/\sqrt n$, so the width falls like $1/\sqrt n$.

To make the half-width at most a chosen margin, solve for $n$:

For a proportion, $\hat p(1-\hat p)$ is never more than $\tfrac14$, its value at $\hat p=\tfrac12$. So the worst case needs

Worked example

1. An opinion poll. Of 1000 people asked, 540 support a plan, so $\hat p=0.54$. The standard error is $\sqrt{0.54\times0.46/1000}=0.01576$, the margin $1.96\times0.01576=0.0309$, and the 95 % interval runs from $0.509$ to $0.571$.

At the worst case $\hat p=0.5$ the margin is $0.0310$: the familiar “plus or minus 3 points” of a poll of 1000. For plus or minus 2 points you need $n=(1.96/(2\times0.02))^2=2401$ people, and for 1 point, $9604$. Halving the margin costs four times the people.

2. Eight response times. The eight times of Describing data have $\bar x=16.25$ ms and $s=2.9155$ ms. The standard error is $2.9155/\sqrt8=1.0308$. With $t_{0.975,\,7}=2.3646$ the margin is $2.437$, and the interval runs from $13.81$ to $18.69$ ms.

Using 1.96 instead would give a margin of $2.02$, too narrow: with only eight values, $s$ is too uncertain to treat as the true spread.

3. A small count. Three of 20 parts fail a test, so $\hat p=0.15$. The plain interval is $0.15\pm0.157$, which runs below 0, an impossible failure rate. The Wilson interval is $0.052$ to $0.360$.

4. Known spread. With $\sigma_X=2$ and $n=25$, the 95 % half-width is $1.96\times0.4=0.784$. With $n=100$ it halves, to $0.392$.

Try it yourself. A large sample with $\bar x=50$, $s=10$ and $n=100$ gives the interval $50\pm1.96$. The same sample gives a wider interval at 99 %: catching the truth more often needs more room.

Where you’ll meet this

Polls and elections. Every published poll carries a margin of error, the half-width of a 95 % interval for a share, usually quoted for the worst case $\hat p=\tfrac12$.

Experiments. An A/B test on a website reports the difference in sign-up rates between two versions with an interval. An interval for the difference that excludes 0 is evidence of a real difference, the question of Significance tests.

Measurements. A lab result such as "$9.81\pm0.02$" is an estimate with an uncertainty. Whether the $\pm$ is one standard error or a 95 % half-width must be stated; it differs by a factor of about 2.

Planning. The sample-size formula decides how many users to survey, patients to enrol or parts to test before the work starts.

Reference card