Models and loss: loss function

A line runs through some points. By how much is it wrong? Below, a line turns through twelve points while one number, its loss, is traced underneath. Before it plays: will the loss be lowest when the line passes through the most points?

How wrong is this line?

Twelve readings, a line through the origin ŷ = wx, and its average squared loss against the slope w.

Twelve readings and a line through the origin, ŷ = wx. Each bar is a residual eᵢ = yᵢ − ŷᵢ, the miss at one reading.

slope w

0.00

average squared loss

20.953

Σ xᵢeᵢ

200.82

Slope, w0.00

Describe this picture

Two panels. Above, twelve readings $y$ against $x$ , with $x$ from 0 to 6.5 and $y$ from −2 to 10, rising from left to right, and a line through the origin, $\hat y=wx$ . A vertical bar joins each reading to the line: its residual, the miss at that reading. Below, the average squared loss is plotted against the slope $w$ , from 0 to 2.5. In the clip the bars appear, then the slope sweeps from 0 up to 2.5 while the lower panel traces the loss; as the slope rises, the bars of the readings above the line shrink and those below it grow. The slope comes back to where the curve is lowest, and the lowest point is marked: $w=1.24$ , average squared loss 0.272, where the bars balance. Readouts give the slope $w$ , the average squared loss and the weighted balance of the bars, $\sum x_ie_i$ , which is 0 at the lowest point. After the clip a slider, “Slope, w”, runs from 0 to 2.5 in steps of 0.01 and starts at 1.24; the line can also be dragged on either panel.

Residuals and two losses

The model here is a line through the origin, $\hat y=wx$ , with one parameter, its slope $w$ . For example $i$ it predicts $\hat y_i=wx_i$ , and the reading is $y_i$ . The difference is the residual,

e_i=y_i-\hat y_i,

the length of the bar, positive when the reading is above the line. I always write the residual with its index, because a bare $e$ is Euler’s number.

Twelve residuals are twelve numbers. A loss turns them into one. The two most common are the average squared residual and the average absolute one:

J_\text{sq}=\frac1m\sum_{i=1}^{m}e_i^2,\qquad J_\text{abs}=\frac1m\sum_{i=1}^{m}\lvert e_i\rvert,

where $m$ is the number of examples, here 12; statistics writes $n$ for a sample size, and on these pages $m$ counts examples. Both are zero only if every reading lies on the line, and both grow as the line moves away. The squared loss punishes one big miss far more than several small ones: a residual of 3 costs 9, three residuals of 1 cost 3.

The loss as a curve

Each slope $w$ gives one value of the loss, so the loss is a function of the parameter, $J_\text{sq}(w)$ , and the lower panel draws it. Expanding the square,

J_\text{sq}(w)=\frac1m\sum_{i=1}^{m}(y_i-wx_i)^2=\frac1m\Big(\sum y_i^2-2w\sum x_iy_i+w^2\sum x_i^2\Big),

a parabola in $w$ that opens upwards. Completing the square puts its lowest point at

w=\frac{\sum x_iy_i}{\sum x_i^2}.

The derivative as a slope finds the same point a second way, where the curve is flat. Fitting the model means choosing the parameter that makes the loss smallest.

Now the answer to the question. The best line need not pass through any point at all. The loss is lowest where the bars balance, but not as a plain count: at the best slope $\sum_ix_ie_i=0$ . Each bar is weighted by its $x$ , because turning the line about the origin moves it most far to the right, so a bar far to the right counts more. Drag the slope and watch the bars on the two sides trade lengths.

In probability and statistics · residuals and least squares

Linear regression in the statistics course draws each squared residual as a square and fits a line with a slope and an intercept the same way. Here the line keeps to the origin so that the loss has one parameter and can be drawn as a curve.

Which loss

The choice of loss changes which model wins. The simplest model makes one prediction, a constant $c$ , whatever the input. Take five readings, 2, 3, 4, 5 and 16, and ask which constant is best.

The best constant depends on the loss

Five readings, one constant c for all of them, and two ways to average the misses.

Five readings: 2, 3, 4, 5, 16. One constant c is the prediction for all of them; each bar is one reading's miss.

constant c

average squared loss

average absolute loss

Describe this picture

Three panels on one axis of the constant $c$ , from 0 to 18. At the top, the five readings 2, 3, 4, 5 and 16, each with a bar to the constant: its miss. Below it, the average squared loss against $c$ , a parabola; at the bottom, the average absolute loss, a bent line. In the clip $c$ slides from 0 to 18 while both curves are traced. Then the lowest points are marked in turn: the squared loss at the mean, 6, where it is 26, and the absolute loss at the median, 4, where it is 3.2. Readouts give the constant $c$ and the two losses. There is a play button and no other control.

Under squared loss the best constant is the mean, 6. Under absolute loss it is the median, 4. One reading, the 16, drags the squared answer above four of the five readings, because a far reading’s squared residual is large. The absolute loss only counts how far each reading is, so the middle reading wins, and the 16 could be 1,600 without moving it.

Neither loss is the right one. Squared loss suits readings whose errors are small and many; absolute loss suits readings with occasional wild values.

In probability and statistics · mean against median

Describing data in the statistics course compares the mean and the median as summaries of a sample, and shows how one outlier moves the mean. They are the best constants under these two losses.

Losses for classes

For a class there is no residual to square. The natural loss is the 0–1 loss: the share of examples given the wrong class. It counts mistakes, which is what one cares about in the end, but it is a poor thing to fit by. Slide a boundary a little and usually no example changes side, so the loss does not change at all; when one does cross, the loss jumps. As a curve against the parameter it is a staircase, flat almost everywhere, and a flat step gives no hint which way is better. Cross-entropy loss builds a smooth loss for classes.

So a loss and the number of mistakes are different measures. The loss is what fitting makes small; the mistakes, or the error in the units of the problem, are what the user sees.

Watch out

A loss of zero on the training examples is not the goal. A rule can fit every training example and still do badly on new ones.

Worked example

Four points, $(1,1)$ , $(2,3)$ , $(3,5)$ and $(4,5)$ , and the line $\hat y=wx$ .

The sums. $\sum y_i^2=1+9+25+25=60$ , $\sum x_iy_i=1+6+15+20=42$ and $\sum x_i^2=1+4+9+16=30$ . With $m=4$ ,

J_\text{sq}(w)=\tfrac14\,(60-84w+30w^2)=15-21w+7.5w^2.

Two guesses. At $w=1$ the predictions are 1, 2, 3, 4, the residuals $0,1,2,1$ and the loss $(0+1+4+1)/4=1.5$ ; the formula gives $15-21+7.5=1.5$ . At $w=2$ the residuals are $-1,-1,-1,-3$ and the loss $(1+1+1+9)/4=3$ .

The best slope. $w=42/30=1.4$ . The residuals are $-0.4$ , $0.2$ , $0.8$ and $-0.6$ , and the loss is $(0.16+0.04+0.64+0.36)/4=1.2/4=0.3$ . The absolute loss there is $(0.4+0.2+0.8+0.6)/4=2.0/4=0.5$ . The weighted bars balance: $1(-0.4)+2(0.2)+3(0.8)+4(-0.6)=0$ .

One more point. Add $(5,20)$ . Then $\sum x_iy_i=42+100=142$ and $\sum x_i^2=30+25=55$ , so the best slope rises to $142/55=2.58$ : one far point turns the whole line.

Best constants. For 2, 3, 4, 5, 16: at the mean, 6, the average squared loss is $(16+9+4+1+100)/5=130/5=26$ , against $150/5=30$ at 4. At the median, 4, the average absolute loss is $(2+1+0+1+12)/5=16/5=3.2$ , against $20/5=4$ at 6. For 1, 2 and 100 the best constant under absolute loss is the median, 2.

Where you’ll meet this

Calibration and least squares. Fitting a sensor’s calibration line, or any least-squares fit, makes the average squared residual small.

Data with outliers. Where occasional wild readings are expected, an absolute loss keeps one bad reading from dragging the fit.

Costs that are not equal. A demand forecast that runs short loses sales, while one that overstocks only ties up stock, so the loss can charge a shortfall more than an excess.

In signal processing · the median against spikes

Smoothing filters in the signal processing course removes spikes with a median filter where a moving average smears them: the moving average is the best constant under squared loss over each window, the median the best under absolute loss.

Reference card

Quantity	Formula	Notes
Residual	$e_i=y_i-\hat y_i$	always with its index
Squared loss	$J_\text{sq}=\frac1m\sum_{i=1}^{m}e_i^2$	punishes big misses most
Absolute loss	$J_\text{abs}=\frac1m\sum_{i=1}^{m}\lvert e_i\rvert$	less moved by outliers
Line through the origin	$J_\text{sq}(w)=\frac1m(\sum y_i^2-2w\sum x_iy_i+w^2\sum x_i^2)$	a parabola in $w$
Best slope	$w=\sum x_iy_i/\sum x_i^2$	where $\sum_ix_ie_i=0$
Best constant	mean (squared), median (absolute)
0–1 loss	share of wrong classes	a staircase, no slope to follow
Fitting	choose the parameters with the smallest loss	judged later on new data

Models and loss

How wrong is this line?

Residuals and two losses

The loss as a curve

Which loss

The best constant depends on the loss

Losses for classes

Worked example

Where you’ll meet this

Reference card

Keep reading with a free account.

Are you 13 or over?

Where to go next.

Account

Subjects

Site

Look

Models and loss

First, the picture

Residuals and two losses

The loss as a curve

Which loss

Losses for classes

Worked example

Where you’ll meet this

Reference card