Covariance and correlation

Do two things rise together? Five points answer it with rectangles. Watch each one draw its rectangle to the means.

Rectangles around the means

Take five pairs of measurements, say a plant’s height and its number of leaves. Plot each pair as a point. Do the two numbers go up together?

Here is a way to measure it. Draw a line at the mean of each number; for these five points both means are 3. Every point now sits in one of four quadrants. A point to the top right is above average in both numbers. A point to the bottom left is below average in both. Both agree, so I count their rectangle as positive. A point to the top left or bottom right disagrees, so its rectangle counts as negative.

The rectangle’s sides are the point’s distances from the two means, so its area is the product of the two deviations, with its sign. For the point $(5,5)$ that is $(5-3)(5-3)=4$. For $(1,2)$ it is $(1-3)(2-3)=2$: both deviations are negative, and their product is positive. The points $(3,4)$ and $(4,3)$ sit on a dashed line, so one side of their rectangle is 0.

Add the five areas, $2+2+0+0+4=8$. A large positive total says the points mostly agree: when one number is high, so is the other. A large negative total says they mostly disagree. A total near 0 says neither wins.

After the clip, drag a point into a disagreeing quadrant and watch the total drop.

Covariance

Write the points as $(x_k,y_k)$ for $k=1,\dots,n$, with means $\bar x$ and $\bar y$. The sum of the signed rectangles is

It grows with the number of points, so I divide by it. The average rectangle, $S_{xy}/n$, is the covariance of the data: $8/5=1.6$ here. Some books divide by $n-1=4$ instead and get $2.0$; the reason belongs to estimation, and it does not change anything on this page.

For two random variables $X$ and $Y$, with means $\mu_X$ and $\mu_Y$, the covariance is the expected rectangle:

The second form comes from multiplying out the brackets. The covariance of $X$ with itself is its variance, from Variance and standard deviation: every rectangle is then a square.

Correlation

Covariance has a problem: it depends on the units. Measure the heights in centimetres instead of metres and every rectangle gets 100 times wider, so the covariance is 100 times larger. The points have not changed at all.

So scale it. Divide by the spread of each number, and the units cancel. The result is the correlation. For data,

where $S_{xx}=\sum_k(x_k-\bar x)^2$ and $S_{yy}=\sum_k(y_k-\bar y)^2$. For the five points, $S_{xx}=4+1+0+1+4=10$ and $S_{yy}=1+4+1+0+4=10$, so $r=8/\sqrt{100}=0.8$. For random variables,

The correlation always lies between $-1$ and $1$. It is $1$ when the points lie on a rising straight line, and $-1$ on a falling one. For $y=2x+1$, $r=1$; for $y=-3x$, $r=-1$. Changing units, centimetres to metres, leaves $r$ alone.

Why can it never pass 1? Collect the deviations into two lists: $(-2,-1,0,1,2)$ for the first numbers and $(-1,-2,1,0,2)$ for the second. Treat each list as an arrow with five entries. Then $S_{xy}$ is their dot product, and $\sqrt{S_{xx}}$ and $\sqrt{S_{yy}}$ are their lengths. So $r$ is the cosine of the angle between the two arrows, and a cosine stays between $-1$ and $1$. Here the angle is 36.87°, which is 0.6435 radians, and its cosine is $0.8$.

Correlation is not the slope of the line through the points. The best line through them rises by $r\,s_y/s_x$ for each step in $x$, where $s_x$ and $s_y$ are the standard deviations of the $x$ and $y$ values. For the five points, the two spreads are equal, so the slope happens to be $0.8$ too. If $r=0.8$ but the spread of $y$ is twice the spread of $x$, the slope is $1.6$.

What correlation misses

Correlation measures only how close the points come to a straight line. Two variables can depend on each other completely and still have correlation 0. Let $X$ be one of $-2,-1,0,1,2$, each with chance $\tfrac15$, and let $Y=X^2$. Knowing $X$ tells you $Y$ exactly. But the points $(-2,4)$ and $(2,4)$ cancel, and so do $(-1,1)$ and $(1,1)$: $\mathbb{E}\{XY\}=\mathbb{E}\{X^3\}=0$ and $\mathbb{E}\{X\}=0$, so the covariance is 0. The points lie on a curve, not a line.

So zero correlation does not mean independent. The other way round does hold: independent variables have covariance 0, because the expected product of independent variables is the product of their means.

Correlation can also point the wrong way. Watch six points, first in one colour, then in two.

All six points together have $r=0.632$: they seem to rise together. Inside each group, they fall together perfectly. A third thing, the group, raises both numbers at once, and that makes the pooled cloud rise. Imagine the groups are two schools, $x$ is hours of homework and $y$ is a test score. In the pooled data more homework goes with a higher score, while inside each school more homework goes with a lower score.

This is why correlation is not cause. A correlation says the numbers move together in this data. It does not say that one moves the other, and a hidden third variable can create or reverse it.

Worked example

1. Five points. $S_{xy}=8$, $S_{xx}=S_{yy}=10$, so $r=8/\sqrt{100}=0.8$. The covariance is $1.6$ dividing by $n=5$, or $2.0$ dividing by $n-1=4$.
2. Dependent but uncorrelated. $X$ equally likely to be $-2,-1,0,1,2$ and $Y=X^2$: covariance 0, though $Y$ is fixed by $X$.
3. Two groups. The six points have $S_{xy}=14$, $S_{xx}=17.5$ and $S_{yy}=28$, so $r=14/\sqrt{490}=0.6325$, although each group alone has $r=-1$.
4. Lines. $y=2x+1$ has $r=1$; $y=-3x$ has $r=-1$.
5. Slope. With $r=0.8$ and the spread of $y$ twice that of $x$, the best line has slope $1.6$, not $0.8$.

Where you’ll meet this

An investor who holds two shares whose prices have a negative correlation tends to lose less on a bad day, because one tends to rise when the other falls. That is the idea behind spreading money over many investments.

A robot that combines two sensors, say a wheel counter and a camera, needs to know how their errors are correlated; errors that move together cannot cancel. And a data analyst choosing which measurements to feed a model often drops one of two measurements that are almost perfectly correlated, because the second adds little.

Reference card