Bias and variance of a model

If you collected the data again, would you get the same curve? Below, one curve is measured 20 times over, each time with fresh noise, and a polynomial is fitted to each set of readings. Before the clip plays: which degree will stay closest to the true curve?

One curve, many training sets

The true curve is $y_\text{true}(x)=2.5(x^3-x)+x^2$ on $-1\le x\le1$. I built it from a cubic on purpose, so that you can see a model match it exactly; real curves rarely oblige. It starts at 1 at $x=-1$, rises to 1.385 at $x=-0.726$, falls to −0.695 at $x=0.459$ and returns to 1 at $x=1$.

Each reading is the curve plus noise: $y_i=y_\text{true}(x_i)+\varepsilon_i$ at the positions $x_i$, $i=1,\dots,12$. The noise $\varepsilon_i$ has mean 0 and standard deviation $\sigma=0.3$, independent from reading to reading. Here $\sigma$ is a noise’s standard deviation, as in statistics; in signal processing a bare $\sigma$ means something else. The 12 readings, one training set, are $m=12$ examples.

A polynomial of some degree is fitted to each set by least squares, as in Least squares:

with $p=\text{degree}+1$ coefficients $a_0,\dots$, counted by $r$. This is linear regression with the features $x, x^2, x^3,\dots$; Features and basis functions builds that idea.

One fit tells me little. The point of the picture is the bundle: the same model fitted to 20 training sets that differ only in their noise. Now the answer to the question. At degree 1 the 20 lines are nearly the same, and all miss the curve in the same places. At degree 6 the fits are centred on the curve, but each bends towards its own dots, and the bundle spreads, most near the ends. Degree 3 is both tight and on the curve. Drag the degree from 0 to 6 and watch the bundle tighten onto the curve, then spread.

Bias and variance of a model

Imagine fitting not 20 training sets but every possible one, and averaging the fits at a position $x$. That average is the expected fit, $\mathbb{E}\{\hat y(x)\}$, the average over training sets. Two things can go wrong.

The expected fit can miss the curve. That miss is the bias of the model at $x$:

The fits can also scatter round their own average. That scatter is the variance of the model at $x$:

Both describe the bundle, not one fit. Variance here is the spread of the fitted curves, not the noise in the readings. And this bias is a property of a model’s average fit; it is not the bias term $b$ of a linear model.

With only 20 fits, the instrument averages what it has. I write $\bar{\hat y}_S(x)$ for the average of the $S=20$ fits at $x$; the bar marks an average. The readout “bias², measured” is the squared miss of that average, and “variance, measured” the spread of the 20 fits about it, each averaged over the 12 positions. The average of only 20 fits still carries a twentieth of the variance in its squared miss, so the measured bias² sits a little above the true one, and the “expected” readout beside it adds that twentieth: $\text{bias}^2+\text{variance}/20$. At degree 3 the true bias is 0, yet the expected readout shows 0.0015, which is $0.03/20$.

Three parts of the error

What matters is the error on a new reading, $y=y_\text{true}(x)+\varepsilon$, with fresh noise $\varepsilon$ of mean 0 and variance $\sigma^2$, independent of the training sets. Its squared error, averaged over training sets and over the new noise, splits into three parts:

Here is why. Write $y-\hat y=\varepsilon+\big(y_\text{true}-\mathbb{E}\{\hat y\}\big)+\big(\mathbb{E}\{\hat y\}-\hat y\big)$. Square it and average. The new noise is independent of the fit and has mean 0, so every cross term with $\varepsilon$ averages to 0. The last bracket also has mean 0, so its cross term with the fixed middle bracket vanishes too. What is left are the three squares: $\sigma^2$, the bias squared, and the variance.

The noise $\sigma^2$ is the floor: no model can predict fresh noise. Bias and variance are the two parts a choice of model can change.

Averaged over the positions

Average each part over the 12 training positions; a bar over a quantity marks that average. Least squares then gives three exact results, when the positions stay fixed and only the noise is redrawn:

Once the bias has gone, each extra coefficient adds $\sigma^2/m$ to the variance, the training error falls by the same amount and the error on new readings rises by it. These are the formulas of Overfitting and model choice, now with a bias term for models that cannot reach the curve.

The U-shaped curve of Overfitting and model choice is the sum of these parts. On the left, bias dominates: the model is too simple. On the right, variance dominates: the model is flexible enough to follow the noise. In between, at degree 3, bias has reached 0 while variance is still small.

More data

More readings shrink one part and not the other. With 120 positions instead of 12, $\sigma^2p/m$ is ten times smaller at every degree. Bias does not change in that way: a straight line cannot follow a cubic however many readings it gets.

So “enough data” cures variance, never bias. And whether a model “overfits” is not a yes-or-no question: it depends on how much data it has, and by how much its error on new readings exceeds the best.

Worked example

Take the 12 positions $x_i=(2i-13)/12$: $\pm\tfrac1{12},\pm\tfrac3{12},\dots,\pm\tfrac{11}{12}$, with $\sigma=0.3$, so $\sigma^2=0.09$.

The sums. Over the 12 positions, $\sum x_i^2=3.972222$, $\sum x_i^4=2.344715$ and $\sum x_i^6=1.632297$.

Odd and even parts. The curve splits into an odd part, $2.5(x^3-x)$, and an even part, $x^2$. On positions symmetric about 0, least squares fits the two separately: odd powers fit the odd part and even powers the even part.

• The odd part left unfitted (degree 0) leaves $\sum(x_i^3-x_i)^2=\sum x_i^6-2\sum x_i^4+\sum x_i^2=0.915089$. After the best slope (degrees 1 and 2) it leaves $\sum x_i^6-(\sum x_i^4)^2/\sum x_i^2=0.248263$. From degree 3 it leaves 0.
• The even part, after the best constant (degrees 0 and 1), leaves $\sum x_i^4-(\sum x_i^2)^2/12=1.029836$. From degree 2 it leaves 0.

Bias². Average over the 12 positions, with $2.5^2=6.25$ on the odd part:

• degree 0: $(6.25\times0.915089+1.029836)/12=0.5624$;
• degree 1: $(6.25\times0.248263+1.029836)/12=0.2151$;
• degree 2: $6.25\times0.248263/12=0.1293$;
• degrees 3 to 6: 0.

The table. Variance is $0.09p/12$, training error $\overline{\text{bias}^2}+0.09(1-p/12)$ and error on new readings $\overline{\text{bias}^2}+0.09(1+p/12)$:

The training error falls at every degree. The error on new readings is lowest at degree 3, 0.12, and each degree after it adds $0.09/12=0.0075$, one more coefficient’s share of the noise. Degree 6 is 0.0225 worse than degree 3. Is it overfitting? It is worse by a measurable amount, which is a better answer than yes or no.

At 120 positions, $x_i=(2i-121)/120$, the variance falls tenfold: degree 6’s is $0.09\times7/120=0.00525$. Bias² does not fall: degree 1’s is 0.2316 and degree 2’s 0.1427, a little above their values at 12 positions and close to their values over the whole interval, $1/7+4/45=0.2317$ and $1/7=0.1429$, which more positions approach. More data cannot remove bias.

The floor. No model does better on new readings than $\sigma^2=0.09$, the noise itself.

Where you’ll meet this

Calibrating a sensor. A thermocouple’s voltage is turned into a temperature by a polynomial. Too low a degree misses the true curve everywhere; too high a degree follows the noise in the calibration readings.

Nearest neighbours. In Nearest neighbours, $k=1$ is the high-variance end: each stray training dot moves the answer. A large $k$ averages many dots and moves towards bias.

Choosing a setting. On real data the true curve is unknown, so bias cannot be measured. Validation and cross-validation estimates the error on new readings directly, Learning curves shows which part more data would help, and Ridge regression trades a little bias for less variance.

Reference card