Estimators: bias and variance

Take 4 points, average them, and fire the average at a target. Repeat 500 times, and watch where the shots land.

An estimate is a draw

Somewhere there is a true value I want: the bull’s-eye. I only see a sample, and from it I compute a number that I hope is close. The true value is a parameter, written $\theta$; the rule that turns a sample into a guess is an estimator, written $\hat\theta$. The hat means “estimated”.

The middle panel shows one sample and one estimate. Draw a new sample and you get a new estimate: an estimator is a random variable, with its own distribution over repeated samples. The third panel draws that distribution, and it is a different thing from both the population and the one sample. Keep the three apart.

Each coordinate of estimator A is the average of 4 draws with standard deviation 1. As in The law of large numbers, such an average has standard deviation $1/\sqrt4=0.5$. So A’s shots scatter about 0.5 each way around the bull’s-eye. B’s shots are A’s times 0.8: they scatter only 0.4 each way, but around $(0.8,0.8)$.

Which is better? A is right on average but scattered. B is off-centre but tight. The readouts say B lands closer, on average: a mean squared distance of 0.4212 against A’s 0.5292, close to the exact 0.40 and 0.50.

Bias, variance and their sum

Two numbers describe an estimator’s cloud of shots. Its bias is how far the centre of the cloud is from the truth:

Its variance $\mathrm{Var}\{\hat\theta\}$ is how widely the shots scatter around their own centre. An estimator with bias 0 is unbiased.

What I care about is how far the shots land from the truth. The mean squared error measures it:

Write $\hat\theta-\theta$ as $(\hat\theta-\mathbb{E}\{\hat\theta\})+(\mathbb{E}\{\hat\theta\}-\theta)$: a scatter part with mean 0, plus the fixed bias. Square and take the mean. The cross term has a factor with mean 0, so it drops out:

For one coordinate of the target: A has bias 0 and variance $0.25$, so its MSE is $0.25$. B has bias $-0.2$ and variance $0.64\times0.25=0.16$, so its MSE is $0.04+0.16=0.20$. The target adds the two coordinates, so its readouts are twice these: $0.50$ for A, and $0.08+0.32=0.40$ for B.

Why divide by $n-1$

Describing data divided the squared deviations by $n-1$ to get $s^2$. Here is an exact count that shows why.

The sum of squared deviations averages 6 over the nine samples. Divide it by $n=2$ and the estimate averages 3, half the truth. Divide by $n-1=1$ and it averages 6, exactly the truth.

The reason holds for every sample size. The deviations are measured from the sample’s own mean $\bar x$, which sits in the middle of the sample and so is closer to the points than the true mean $\mu_X$ is. Exactly how much closer is an identity:

Here the $X_k$ are the draws as random variables, before the sample is seen. Take means. The first sum has mean $n\sigma_X^2$. The average $\bar X_n$ has variance $\sigma_X^2/n$, so the last term has mean $\sigma_X^2$. What is left is $(n-1)\sigma_X^2$, so

With divisor $n$ the mean is $\frac{n-1}{n}\sigma_X^2$, a bias of $-\sigma_X^2/n$. For $n=10$ that is $-\sigma_X^2/10$.

Shrinking: a little bias for less scatter

Estimator B multiplied the average by a shrink factor $c$, here 0.8. In general, for the average $\bar X_n$ of $n$ draws,

The first term is the variance, which shrinks with $c$; the second is the bias squared, which grows as $c$ moves away from 1. Their sum is smallest at

For the target, $\mu_X=1$, $\sigma_X=1$ and $n=4$, so the best factor is $1/1.25=0.8$. After the clip, drag $c$ and watch the total: $0.625$ at $c=0.5$, $0.40$ at $0.8$, $0.50$ at 1 and $0.80$ at $1.2$, a U with its bottom at 0.8.

There is a catch. The best $c$ depends on $\mu_X$, the very thing being estimated. In practice it is chosen from earlier data or a rule, and the gain is real only when the guess is reasonable. The lesson stands, though: an unbiased estimator is not always the best one.

Consistency

An estimator is consistent if it settles on the truth as the sample grows; an MSE that falls to 0 is enough to show it. The average $\bar X_n$ is unbiased with variance $\sigma_X^2/n$, so its MSE falls to 0. With $\sigma_X=2$ and $n=16$ it is already $4/16=0.25$.

Worked example

1. The nine samples. Over the nine samples the divisor-$n$ estimate averages $54/9/2=3$, which is $\sigma_X^2/2$, half the truth. The divisor-$(n-1)$ estimate averages $6=\sigma_X^2$.

2. Shrinking a mean. With $\mu_X=1$, $\sigma_X=1$ and $n=4$, the average $\bar X$ has MSE $0.25$. The estimate $0.8\bar X$ has bias $-0.2$, variance $0.16$ and MSE $0.20$.

3. Three divisors for a variance. Take bell-shaped data with $\sigma_X^2=1$ and $n=5$, and divide the sum of squared deviations by 4, 5 or 6:

The unbiased divisor, 4, has the largest error. Dividing by 6 is biased low but lands closest on average. Which one to use depends on what you need: right on average, or close on average.

Where you’ll meet this

Calibration. A sensor is calibrated by averaging repeated readings of a known reference; the bias of the average is the calibration error, and its variance falls as $1/n$.

Regression. Shrinkage estimators such as ridge regression pull fitted coefficients towards zero: a little bias for much less scatter, the trade of the target.

Experiments. Every A/B test reports an estimate, and its standard error is the square root of the estimator’s variance, the subject of Confidence intervals.

Reference card