Sums of random variables

Throw two dice and add them. Why is 7 the most likely total? Watch one die’s chances slide across the other’s.

Counting the ways

Why does 7 come up more often than 2? There is only one way to throw 2: a 1 and a 1. There are six ways to throw 7: 1 and 6, 2 and 5, 3 and 4, 4 and 3, 5 and 2, 6 and 1. Each of the 36 pairs has chance $\tfrac1{36}$, so $\Pr\{7\}=\tfrac6{36}=\tfrac16$, and $\Pr\{2\}=\tfrac1{36}$.

The picture does the same count with the chances themselves. For a total $T$, the first die shows some $k$ and the second shows $T-k$. If the dice are independent, the chance of that pair is $p_X(k)\,p_Y(T-k)$. Add over every $k$:

That sum is a convolution, written $p_{X+Y}=p_X*p_Y$. The second die is reversed because as $k$ goes up, $T-k$ goes down. After the clip, drag the total $T$ to 4: three pairs line up, $(1,3)$, $(2,2)$ and $(3,1)$, so $\Pr\{4\}=\tfrac3{36}$.

Means add, always

The mean of a sum is the sum of the means:

This holds whether or not $X$ and $Y$ are independent, because an average of sums is a sum of averages. Two dice have mean $3.5+3.5=7$, the peak of the triangle.

Variances add, if uncorrelated

Spreads are different. Square the distance of the sum from its mean, and multiply out:

Take the mean of each part. The first two are the variances, and the last is twice the covariance of Covariance and correlation:

When $X$ and $Y$ are uncorrelated, and in particular when they are independent, the covariance is 0 and the variances add. Two dice have variance $\tfrac{35}{12}+\tfrac{35}{12}=\tfrac{35}{6}=5.833$, so the total has spread $2.415$.

The correlation matters. Take two variables, each with spread 1. If $\rho_{XY}=0.5$, the sum has variance $1+1+2\cdot0.5=3$. If $\rho_{XY}=1$, they always move together and the variance is 4. If $\rho_{XY}=-1$, each cancels the other and the variance is 0: the sum never moves.

Spreads add as squares

So spreads do not add; variances do. Two independent errors with spreads 3 and 4 give a total with variance $9+16=25$, so a spread of 5, not 7. It is the right-angle rule for the sides of a triangle, with spreads as the sides.

The same goes for a difference. $\mathrm{Var}\{X-Y\}=\mathrm{Var}\{X\}+\mathrm{Var}\{Y\}-2\,\mathrm{Cov}\{X,Y\}$, so for independent variables the variances still add: the difference of two independent unit-variance variables has variance 2, not 0.

And it explains a fact about dice. Adding a second die doubles the variance, so the spread grows only by $\sqrt2$, about 1.41 times. That slow growth is where the $\sigma_X/\sqrt n$ of The law of large numbers comes from.

How errors spread

A measured length is $20.0\pm0.1$ cm and a width is $10.0\pm0.1$ cm, where each $\pm0.1$ is a spread and the two errors are independent. What is the spread of the area, $20\times10=200$ cm²?

A small error $\varepsilon$ in the length changes the area by about $10\,\varepsilon$, because the area is length times 10. A small error in the width changes it by about $20$ times that error. The two changes are independent, so their variances add:

In general, if a result $f$ changes by $c_k$ for each unit change in input $k$, and the inputs have independent errors with spreads $\sigma_k$, then $\sigma_f^2\approx\sum_k c_k^2\sigma_k^2$. This is called propagation of error. It holds well when the errors are small.

Worked example

1. Two dice. Mean 7, variance $35/6=5.833$, spread $2.415$, $\Pr\{7\}=\tfrac16$.
2. A total of 4. $\Pr\{4\}=\tfrac3{36}$.
3. Correlated parts. Spreads 1 and 1: variance of the sum 3 at $\rho_{XY}=0.5$, 4 at $\rho_{XY}=1$, 0 at $\rho_{XY}=-1$.
4. Spreads 3 and 4, independent. The sum has spread 5, not 7.
5. A difference. For independent $X$ and $Y$ with variance 1 each, $\mathrm{Var}\{X-Y\}=2$.
6. A rectangle. $20.0\pm0.1$ by $10.0\pm0.1$: area 200, spread $\sqrt5=2.236$.

Where you’ll meet this

An amplifier with two independent noise sources has a total noise power equal to the sum of the two powers, because power is a variance. A machine part built from a stack of pieces, each with a small error in thickness, has a total error whose spread grows like the square root of the number of pieces, not in proportion to it. Engineers rely on this when they set tolerances.

Reference card