The central limit theorem

Average 30 draws from a lopsided shape, 2000 times. What shape does the pile of averages take? Pick a sketch, then watch.

One draw, one sample, many averages

What did you pick? Many people pick the lopsided sketch, because the draws are lopsided. Others pick the bell, because they have heard that “data is bell-shaped”. The pile at n = 30 is a bell, but not because the data became one.

Look at the top panel again. It is the shape every single draw comes from, and it never moves. I chose an exponential shape, the kind that describes waiting times: most waits are short, a few are long, and none is below zero. Its curve is $f_X(x)=e^{-x}$ for $x\ge0$. Here $X$ is one draw, and $f_X$ is its density, as in Continuous random variables. This shape is called $\mathrm{Exp}(1)$; the 1 is its rate, one arrival per unit of time on average. Its mean is $\mu_X=1$ and its standard deviation is $\sigma_X=1$; the subscript names the random number they belong to. A bare $\mu$ or $\sigma$ appears only inside a bell’s name, $\mathcal{N}(\mu,\sigma^2)$, and signal processing pages use a bare $\sigma$ for something else.

The middle panel is one sample: $n$ draws, and their average. The bottom panel collects $m=2000$ such averages, one sample at a time. Only the bottom panel changes when $n$ grows. The draws stay lopsided. One sample stays a scatter of lopsided draws. It is the pile of averages that rounds into a bell.

So keep the three things apart: the shape of one draw, one sample of $n$ draws, and the pile of many averages. Mixing them up is a common mistake with this theorem. After the clip, drag $n$ from 1 to 50 and watch only the bottom panel.

How wide the pile is

Why does the pile narrow? Write the average of $n$ draws as

The bar marks an average of draws, and $\bar X_n$ is random: a new sample gives a new value. Each draw has mean $\mu_X$, so the average has mean $\mu_X$ too, $(\mu_X+\cdots+\mu_X)/n=\mu_X$. The pile stays centred on 1.

For the spread, I use the rule from Sums of random variables: for draws that do not affect one another, the variances add. The sum of $n$ draws has variance $n\sigma_X^2$. Dividing by $n$ divides the variance by $n^2$. So

This is the spread you met in The law of large numbers. With $\sigma_X=1$, the predicted spreads at the clip’s five steps are:

The readout “spread of the averages” is measured from the 2000 averages on screen, so it wobbles a little around these values. Four times as many draws halve the spread: that square root is why.

The bell

The spread is only half of the story. The surprise is the shape. As $n$ grows, the pile looks more and more like the bell of The normal distribution, whatever shape the draws came from.

To compare shapes I need one scale. Subtract the mean and divide by the spread:

Now $Z_n$ always has mean 0 and spread 1. The central limit theorem says that if the draws are independent, all come from one shape, and that shape has a finite standard deviation $\sigma_X$, then as $n$ grows

where $\Phi$ is the distribution function of the standard bell $\mathcal{N}(0,1)$. In words: the average is close to $\mathcal{N}(\mu_X,\sigma_X^2/n)$, a bell with the same centre and spread as the pile. That is the dotted bell in the picture. The sum of the draws, which I’ll write $S_n=X_1+\cdots+X_n$ (a sum, not the sample space $S$), is $n$ times the average, so it is close to $\mathcal{N}(n\mu_X,\,n\sigma_X^2)$.

This is not the law of large numbers again. That law says the average settles at $\mu_X$, so the pile shrinks towards one spike. This theorem says what shape the pile has on the way, once you zoom in by $\sqrt n$.

How fast the bell arrives

How large must $n$ be? It depends on how lopsided the start is. One measure of lopsidedness is the skewness: zero for a shape that is the same on both sides, positive when a long tail runs to the right. The exponential shape has skewness 2. The average of $n$ of its draws has skewness $2/\sqrt n$, so the lean fades, but slowly.

At n = 30 the skewness is still 0.37, so the pile in the picture leans a little to the right, even though it looks like a bell. The worked example below shows what that lean costs in a tail.

Counting heads with a bell

Toss a fair coin $n$ times and count the heads. The count is a sum of $n$ draws, each 1 for a head and 0 for a tail. Each draw has mean $\tfrac12$ and standard deviation $\tfrac12$. So the count is close to a bell with mean $n/2$ and variance $n/4$, as the sum rule above says.

In general the count $K$ of successes in $n$ tries with chance $p$ follows $\mathrm{Bin}(n,p)$, from Discrete random variables, and

A usual rule of thumb asks for $np\ge10$ and $n(1-p)\ge10$ before you trust it.

There is one catch. The count is a whole number, drawn as bars, and the bell is a smooth curve. Watch the bars of $\mathrm{Bin}(n,\tfrac12)$ and the bell together, with the bars at or above $0.6n$ filled.

Each bar is one unit wide: the bar for 60 heads stands on the stretch from 59.5 to 60.5. So “60 heads or more” is the area of the bars from 59.5 upwards, and the bell’s area should start there too, not at 60. Starting half a bar early is the continuity correction:

After the clip, step n through its values and compare the three readouts. The half-bar bell stays close to the exact chance even at n = 20; the bell without it is far off there.

Worked example

Waiting at a counter. Service times at a counter follow an exponential shape with mean 2 minutes, so the standard deviation is 2 minutes too. What is the chance that 36 customers average more than 2.5 minutes?

1. The average of 36 has mean 2 and spread $2/\sqrt{36}=2/6=0.3333$ minutes.
2. Standardise: $z=(2.5-2)/0.3333=1.5$.
3. By the bell: $\Pr\{\bar X_{36}>2.5\}\approx1-\Phi(1.5)=0.0668$.
4. Computed exactly, without the bell, the chance is 0.0742. The bell is too low by about a tenth.
5. Why too low? The averages still lean right: their skewness is $2/\sqrt{36}=0.33$. A lean to the right means a heavier right tail than the bell’s.

Sixty heads. Toss a fair coin 100 times. What is the chance of 60 heads or more?

1. The mean is $100\cdot\tfrac12=50$ and the spread is $\sqrt{100\cdot\tfrac12\cdot\tfrac12}=5$.
2. Exactly, from the binomial: 0.0284.
3. Bell with the half-bar: $z=(59.5-50)/5=1.9$, and $1-\Phi(1.9)=0.0287$.
4. Bell without it: $z=(60-50)/5=2$, and $1-\Phi(2)=0.0228$.
5. The same question at 20 tosses, 12 heads or more: exactly $263\,950/2^{20}=0.2517$; with the half-bar, $z=1.5/\sqrt5$ and the bell gives 0.2512; without it, $z=2/\sqrt5$ and the bell gives 0.1855.

Try these before you read the answers.

• The average of 25 draws from $\mathrm{Exp}(1)$: what is its spread? $1/\sqrt{25}=0.2$.
• How many draws would halve that spread? Four times as many, 100.
• Do the 30 draws of one sample look like a bell? No. Only their average does; the top panel never moves.

Where you’ll meet this

An opinion poll asks 1000 people a yes-or-no question. The share who say yes is an average of 1000 draws of 0 or 1, so it is close to a bell. Its spread is $\sqrt{p(1-p)/1000}$, where $p$ is the true share, and that is largest at an even split: $\sqrt{0.25/1000}=0.016$. About 95 % of a bell lies within 1.96 spreads of its centre, which is where the familiar “plus or minus 3 points” comes from.

A factory weighs five bags each hour and plots their average. Because the average of five is close to a bell, with a spread $\sqrt5$ times smaller than one bag’s, the chart can warn when the average leaves its usual band, long before single bags look strange.

The hiss of an amplifier is the sum of very many tiny, independent pushes from moving electrons. That is why electronic noise follows a bell so closely, and why noise in signal processing is usually modelled as Gaussian, which is another name for the bell.

The page Confidence intervals uses exactly this spread, $\sigma_X/\sqrt n$, and this bell.

Reference card