Linear regression

Five points and a line. Each point’s gap to the line is drawn as a square. Watch the squares shrink as the line tilts.

Residuals and their squares

The vertical gap from a point to the line is its residual: how far the line misses that point, measured in the units of $y$. Points above the line have positive residuals, points below negative ones.

The least-squares line is the line that makes the total area of the squares, the sum of the squared residuals, as small as possible. Watch the readout as the line tilts: 6.0 when flat, falling to 2.4 at the best line. Move the line anywhere after the clip; no position does better than 2.4.

With the line held through the point $(3,4)$ and tilted to slope $b$, the total area is $6-12b+10b^2$: 6.0 at slope 0, 3.3 at 0.3, 2.4 at 0.6, 4.0 at 1 and 6.0 again at 1.2. It is a parabola in $b$, lowest at $b=0.6$.

The best line

Write the line $y=a+bx$, with intercept $a$ and slope $b$. As in Covariance and correlation, let

Setting the slopes of the sum of squares to zero, in $a$ and in $b$, gives the best line:

The second equation says the best line always passes through the centre of the cloud, $(\bar x,\bar y)$. That is why the clip could tilt the line about $(3,4)$.

The slope can be written with the sample correlation $r$ and the two sample standard deviations $s_x$ and $s_y$:

The fitted values are $\hat y_k=\hat a+\hat bx_k$ and the residuals $\hat\varepsilon_k=y_k-\hat y_k$; I use $\hat\varepsilon$, not $e$, which is Euler’s number here. The residuals of the best line always add to 0.

How much is explained: $R^2$

The spread of $y$ about its mean is $S_{yy}$. What the line leaves over is the residual sum of squares, $\mathrm{SS}_\text{res}=\sum_k\hat\varepsilon_k^2$. Their ratio says how much of the spread the line failed to explain, so

is the share it did explain. For a straight line it equals the squared correlation, $r^2$, which is where the name comes from.

How well the slope is known

Five points pin a line down only roughly: another five points from the same source would give another slope. To say how roughly, assume each $y_k$ is a true line plus an independent bell-shaped error $\varepsilon_k$ of a fixed spread.

Estimate that spread from the residuals, dividing by $n-2$ because two numbers, $\hat a$ and $\hat b$, were fitted:

Here $s$ is the residual standard deviation. The slope’s standard error is $s/\sqrt{S_{xx}}$, and a 95 % interval for the true slope is

with $t_{0.975,\,n-2}$ from t-tests. The line’s height at a point $x_0$ has its own interval, $\hat y_0\pm t_{0.975,\,n-2}\,s\sqrt{\tfrac1n+(x_0-\bar x)^2/S_{xx}}$. It is narrowest at $\bar x$ and widens away from it: the further you predict from the data, the less you know.

Anscombe’s four data sets

A line and its $R^2$ say nothing about whether a line was the right model.

Same line, same $R^2$, four different stories. One number cannot replace the picture: always plot the data and the residuals.

Worked example

1. The five points. $\bar x=3$ and $\bar y=4$. The deviations of $x$ are $-2,-1,0,1,2$ and of $y$ are $-2,0,1,0,1$; their products are $4,0,0,0,2$. So $S_{xx}=10$, $S_{xy}=6$, $S_{yy}=6$, and

The correlation is $r=6/\sqrt{60}=0.7746$, and $r\,s_y/s_x=0.7746\times1.2247/1.5811=0.6$, the same slope.

The fitted values are $2.8,\ 3.4,\ 4.0,\ 4.6,\ 5.2$, and the residuals $-0.8,\ 0.6,\ 1.0,\ -0.6,\ -0.2$, which add to 0. Their squares add to $0.64+0.36+1+0.36+0.04=2.4$, so $R^2=1-2.4/6=0.6$.

2. The slope’s uncertainty. $s^2=2.4/3=0.8$ and $s=0.8944$. The slope’s standard error is $0.8944/\sqrt{10}=0.2828$. With $t_{0.975,\,3}=3.1824$, the 95 % interval for the slope is $0.6\pm0.90$, from $-0.30$ to $1.50$. It includes 0: five points do not pin the slope down. The matching test of slope 0 gives $t_\text{obs}=0.6/0.2828=2.121$ with 3 degrees of freedom and a two-sided $p_\text{val}$ of $0.124$.

3. The band. At $x_0=3$ the line’s height is $4.0\pm1.27$. At $x_0=6$, outside the data, it is $5.8\pm2.99$: more than twice as wide.

4. The same fit by matrices. The two conditions for the best line are the normal equations of Least squares:

with the sums $\sum1=5$, $\sum x_k=15$, $\sum x_k^2=55$, $\sum y_k=20$ and $\sum x_ky_k=66$. Their solution is $(2.2,\ 0.6)$.

Try it yourself. With $r=0.5$, $s_y=4$ and $s_x=2$, the slope is $0.5\times4/2=1$. The best line predicts $2.2+0.6\times6=5.8$ at $x=6$. An $R^2$ of 0.6 does not mean 60 % of the points lie on the line; it means the line accounts for 60 % of the spread of $y$.

Where you’ll meet this

Calibration. A sensor is calibrated by fitting its readings against a reference: the slope is its gain and the intercept its offset.

Trends. A trend line through monthly sales, temperatures or response times is a least-squares fit, often with the band drawn to show how far it can be trusted.

Prediction. Load forecasts, house-price estimates and growth charts start from regression lines, and their bands widen exactly where predictions are wanted most: away from the data.

Reference card