Regression to the mean

A thousand students take two tests. Light up the top tenth on the first, and watch where they land on the second.

The top tenth fall back

Before the clip you guessed. The top tenth on test 1 average about 675 there; on test 2 the same students average about 605, well above 500 but well below where they were. They fell back partway, and nothing was done to them. The clip’s 1000 students came out a little lower on both, 662.8 and 588.9, with the same fall back.

Here is why. A test score is partly lasting skill and partly the luck of the day. The top tenth on test 1 are mostly skilled people who also had a good day. On test 2 their skill stays, but their luck is new, and on average ordinary. So they come back towards the middle. This drift is called regression to the mean.

The line in standard units

Put both scores in standard units, $Z_X=(X-\mu_X)/\sigma_X$ and $Z_Y=(Y-\mu_Y)/\sigma_Y$, each with mean 0 and standard deviation 1. The regression line of Linear regression then has slope $\rho_{XY}$, the correlation, and passes through the origin:

For bell-shaped scores the retests of people with first score $z$ also scatter, with standard deviation $\sqrt{1-\rho_{XY}^2}$.

Since $\rho_{XY}$ is less than 1, the predicted retest is always nearer the mean than the first score: a student 2 standard deviations up is expected $2\rho_{XY}$ up next time. After the clip, drag the correlation. At 1 nobody moves: the second test repeats the first. At 0 the lit group falls all the way back to 500: the first test told nothing about the second.

The top tenth in numbers. The top tenth of a standard bell starts at $z_{0.90}=1.2816$, and its mean is the bell’s height there over 0.1: $f_Z(1.2816)/0.1=1.7550$. So their first-test mean is $500+100\times1.7550=675.50$, and their expected retest mean is $500+175.50\,\rho_{XY}$: about 553 at $\rho_{XY}=0.3$, 605.30 at 0.6, about 658 at 0.9.

From test 2 back to test 1

If the fall back were caused by time, by practice or tiredness, it would only run forwards. It runs both ways.

The top scorers on test 2 also scored lower, on average, on test 1. By symmetry, by the same amount. The drift is a property of picking extremes on a noisy measure, not of anything that happened between the tests.

Regression and treatments

Blood pressure readings have mean 130 and standard deviation 15 in a population, and two readings of the same person correlate at $\rho_{XY}=0.7$. Patients whose first reading was 160, two standard deviations up, are expected at $130+0.7\times2\times15=151$ on the next reading: a fall of 9 with no treatment at all.

Give those patients a pill between the readings, and the pill will seem to work. The only fair comparison is a control group: equally extreme patients, chosen the same way, who do not get the pill. Their fall measures regression to the mean, and only the extra fall in the treated group is the pill’s.

The regression line and the SD line

In standard units, the line through the cloud’s long axis, the SD line, has slope 1: it moves one standard deviation in $y$ for each one in $x$. The regression line has slope $\rho_{XY}$, flatter. The SD line looks like the natural fit, but predicting from it overestimates every extreme case: it ignores the luck that will not repeat.

Worked example

1. Retests. Scores have mean 100, standard deviation 15 and $\rho_{XY}=0.6$. A first score of 130 ($z=2$) predicts a retest of $100+0.6\times2\times15=118$, with spread $15\sqrt{1-0.36}=12$; so 95 % of such retests lie within about $118\pm23.5$. A first score of 70 predicts 82.

2. The top tenth. First-test mean 675.50; retest means $500+175.50\,\rho_{XY}$: 500 at $\rho_{XY}=0$, 605.30 at 0.6, 675.50 at 1.

3. Blood pressure. First reading 160, expected second reading 151, with no treatment.

4. Two lines. With equal spreads and $\rho_{XY}=0.6$, the SD line has slope 1 and the regression line 0.6.

Try it yourself. A first $z$ of $-1.5$ with $\rho_{XY}=0.5$ predicts a retest $z$ of $-0.75$. If you pick the bottom tenth, their average rises on retest. With $\rho_{XY}=1$, every score repeats exactly.

Where you’ll meet this

Sport. The “rookie of the year” who disappoints in the second season, and the team that slumps after a record year, are often regression to the mean: the best first seasons had the most luck in them.

Schools and tests. Schools picked for help because of a bad year often improve the next, help or no help. Only a comparison with similar schools that got no help shows what the help did.

Road safety. Speed cameras placed at sites with many crashes last year are followed by fewer crashes, partly because last year was unusually bad.

Measurements. An extreme sensor reading is partly noise, so the best estimate of the true value lies nearer the average: the idea behind the shrinkage of Estimators: bias and variance and the posterior mean of Priors, posteriors and credible intervals.

Reference card