Joint distributions

Two clouds of points fall, and each casts a shadow on its two axes. Watch the shadows, then look at the clouds.

A cloud and its shadows

So far each random variable has been alone. But measurements often come in pairs: a person’s height and weight, the temperature and the number of ice creams sold, the two parts of a noisy radio signal. To describe a pair I need both numbers at once.

Draw each pair as a point, and many pairs make a cloud. In the picture at the top of the page, the left cloud is round and the right cloud is tilted. In the tilted one, a large first number usually comes with a large second number.

Now look at the edges. The histogram under each cloud counts only the first numbers, as if the cloud cast a shadow straight down. The one at the side counts only the second numbers. Those shadows are the marginal distributions, and in the picture all four are the same bell.

So the shadows do not tell you the cloud. Two very different pairs of variables can have exactly the same marginals. To know how two variables behave together, you need the whole cloud, the joint distribution.

Tables first

The cleanest way to see this is with a table. Let $X$ be 0 or 1 and $Y$ be 0, 1 or 2. The joint distribution gives the chance of each pair, $p_{XY}(x,y)=\Pr\{X=x,\ Y=y\}$:

The six chances add up to 1. The shadows are the totals at the edges. Add along a row to get the chance of each value of $X$, and down a column for $Y$:

So $\Pr\{X=1\}=0.60$ and $\Pr\{Y=2\}=0.50$. For variables with a density, like the clouds above, the same idea holds with a joint density $f_{XY}(x,y)$ and integrals in place of sums.

A slice: the conditional distribution

Now suppose you learn that $X=1$. Only the second row is still possible. Its chances add up to 0.60, not 1, so divide each by 0.60:

Given $X=1$, $Y$ is 0, 1 or 2 with chances $0.05/0.6=0.0833$, $0.15/0.6=0.25$ and $0.40/0.6=0.6667$. That is the conditional distribution of $Y$ given $X=1$, the rule of Conditional probability applied to every value at once. Its mean is $0\cdot0.0833+1\cdot0.25+2\cdot0.6667=1.5833$. Given $X=0$, the chances are $0.25$, $0.5$ and $0.25$, and the mean is $1.0$.

In the cloud, a condition is a slice. After the clip, drag the slice to $x=1.5$. In the round cloud, the points in the slice are spread around 0, as if you had learned nothing. In the tilted cloud they sit higher, around 1.2, and closer together. In the clip’s 1000 points, the slice holds 53, with means $-0.019$ and 1.192 and spreads 0.934 and 0.592.

The mean of a slice, $\mathbb{E}\{Y\mid X=x\}$, is the conditional mean. It is the best guess of $Y$ once you know $X$: of all guesses that depend only on $x$, it gives the smallest average squared miss. Averaging the slice means with the chances of each slice gives back the overall mean: $0.4\cdot1.0+0.6\cdot1.5833=1.35$, which is $\mathbb{E}\{Y\}=0\cdot0.15+1\cdot0.35+2\cdot0.50$.

For the clouds in the picture, both variables have mean 0 and spread 1, and the tilt is set by a number $\rho_{XY}$, the correlation, which the page Covariance and correlation defines: 0 on the left and 0.8 on the right. For clouds of this kind, the bivariate normal, a slice at $x$ has

With $\rho_{XY}=0.8$ and $x=1.5$, the mean is $0.8\cdot1.5=1.2$ and the spread is $\sqrt{1-0.64}=0.6$.

Independence is a product

Two variables are independent, as in Independence, when learning one tells you nothing about the other. In a table, that means every slice has the same shape as the shadow. Equivalently, every cell is the product of its row total and its column total:

The table above fails at the first cell: $0.10$, but $0.40\cdot0.15=0.06$. So $X$ and $Y$ are dependent. Here is the table that has the same totals and is independent:

Same shadows, different joint distributions: the table version of the two clouds.

Worked example

1. A shadow. $\Pr\{Y=2\}=0.10+0.40=0.50$.
2. A slice. $\Pr\{Y=2\mid X=1\}=0.40/0.60=0.6667$.
3. Dependent. $\Pr\{X=0,Y=0\}=0.10$, but $\Pr\{X=0\}\Pr\{Y=0\}=0.06$.
4. Conditional means. Given $X=1$, the mean of $Y$ is $1.5833$; given $X=0$, it is $1.0$; overall, $0.4\cdot1.0+0.6\cdot1.5833=1.35$.
5. The twin table. With the same totals and independence, the rows are $0.06,\ 0.14,\ 0.20$ and $0.09,\ 0.21,\ 0.30$.
6. A cloud slice. With $\rho_{XY}=0.8$, both spreads 1 and both means 0, the slice at $x=1.5$ has mean $1.2$ and spread $0.6$.

Where you’ll meet this

A radio receiver’s noise has two parts, in phase and in quadrature, and their joint distribution is a round cloud like the left one. Two sensors that measure the same room share a tilted cloud. Neighbouring pixels in a photograph are a tilted cloud too, which is why one pixel predicts the next so well, and why images compress.

Reference card