The bootstrap

You have eight response times and no way to get more. Draw from them again, with replacement, a thousand times, and watch their mean wobble.

Resampling a single sample

To see how much a statistic wobbles, I would like many samples from the population. I have one. The bootstrap pretends the sample is the population, and draws new samples from it: $n$ values, chosen at random with replacement, so a value can be picked twice or not at all. Each such draw is a resample.

Before the clip you guessed how many different values a resample holds. About 5 of the 8, on average. Each value is missed by all 8 draws with chance $(7/8)^8=0.344$, so about a third of the values are absent from any one resample, and $8\times(1-0.344)=5.25$ are present. For large samples the missing share approaches $e^{-1}=0.368$.

The resamples are not new data. They hold nothing but the same eight values, rearranged. What they show is how much a statistic would change if the sample had come out a little differently.

The spread of the resample means

Compute the mean of each resample: $\hat\theta^*_1,\dots,\hat\theta^*_m$ for $m$ resamples, the star marking “computed from a resample”. Their histogram is centred on $16.25$, the mean of the sample, not on the unknown true mean. The bootstrap cannot see past the sample.

The spread of the resample means is the bootstrap standard error:

After the clip, drag the number of resamples. The readout wanders with few resamples and settles near $0.964$ ms as they grow; the clip’s 1000 resamples give $0.9326$. That limit is the spread of the sample itself, computed with divisor $n$, $2.7272$, divided by $\sqrt8$. The formula of Confidence intervals, $s/\sqrt n$ with divisor $n-1$, gives $1.0308$: slightly larger, for the reason Estimators: bias and variance gave.

More resamples make the readout steadier, but they never make the sample bigger. A thousand resamples still carry only the information in eight values.

A percentile interval

The resample means also give an interval with no formula at all. Sort them and cut off the lowest 2.5 % and the highest 2.5 %: what is left is a 95 % percentile interval. For the eight times the clip’s 1000 resamples give 14.5 to 18.125 ms, roughly $16.25\pm1.8$; with many more resamples it settles near 14.4 to 18.1.

The t interval of Confidence intervals for the same data was 13.81 to 18.69 ms, a little wider: with eight values, the bootstrap underestimates the spread, as the divisor above suggests.

Statistics with no formula

For a mean, the formula $s/\sqrt n$ already does the job. The bootstrap earns its keep where no simple formula exists: the median, a ratio, a correlation, a trimmed mean. The recipe is the same: compute the statistic on each resample, then read the spread and the percentiles off the histogram. For the median of the eight times, the bootstrap standard error is about 1.3 ms.

When it fails

Take the maximum of eight values. A resample contains the sample’s maximum with chance $1-(7/8)^8=0.656$, so about two resamples in three report exactly that value. The bootstrap histogram of the maximum is mostly one tall spike, while the true distribution of a sample maximum over new samples has no spike at all. For large samples the spike holds about $1-e^{-1}=0.632$ of the resamples.

Worked example

1. Every resample of a tiny sample. The sample $1,2,6$ has $3^3=27$ equally likely ordered resamples. Their means take these values:

The mean of these 27 means is 3, the sample mean. Their variance is $14/9=1.556$, standard deviation $1.247$. That equals the sample’s spread with divisor $n$, $\sqrt{14/3}$, divided by $\sqrt3$: the bootstrap standard error of a mean is always the plug-in spread over $\sqrt n$.

2. The eight times. About 34 % of the values are missing from each resample and 5.25 are present. There are $8^8=16\,777\,216$ ordered resamples, but only $\binom{15}{8}=6435$ different collections of values. The bootstrap standard error of the mean tends to $0.964$, a little below the formula’s $1.031$.

3. The maximum. At $n=8$ a resample repeats the sample maximum with chance $0.656$; the bootstrap is not to be trusted for it.

Try it yourself. Draw 8 from 8 with replacement: a given value is missing with chance $0.344$. The bootstrap histogram of the mean centres on the sample mean.

Where you’ll meet this

Standard errors without formulas. Medians, ratios of two means, correlations and the coefficients of complicated models often get their error bars from the bootstrap in everyday statistics software.

Experiments on skewed data. Revenue per visitor is lopsided: most visitors spend nothing, a few spend a lot. Online A/B tests often bootstrap the difference in mean revenue instead of trusting a bell.

Curves and peaks. The uncertainty of a fitted curve, or of the position of a peak in a spectrum, is found by refitting to resampled data and watching the result move.

Reference card