Independence

A fair coin has just shown five heads in a row. Is tails now due? Guess, then watch 10 000 flips.

Flips after a streak

Before the clip you guessed. Many people feel that after five heads, tails is due: the coin has to even things out. The tally says no. Of the flips that came straight after five heads, about half were heads.

How many such flips are there? Each of the 9995 flips from the sixth on has five flips before it, and those five are all heads with chance $\left(\tfrac12\right)^5=\tfrac1{32}$. So about $9995/32=312$ flips land in the tally; in the clip, 318 did. Inside a long run of heads, several flips in a row count.

With about 312 flips, the typical miss of What probability measures says the share of heads should land within about $\sqrt{0.25/312}=0.028$ of one half. The tally’s share does: 157 heads and 161 tails, a share of 0.494.

After the clip, drag the streak length. After one head, about 5000 flips land in the tally and the share hugs one half, within about 0.007. After seven heads, only about 78 flips land there and the share wobbles more, about 0.057; in the clip’s flips, 83 landed there, 47 of them heads, a share of 0.566. But it wobbles around one half, with no lean towards tails.

Independent events

The tally measured a conditional probability. With $A$ “this flip is heads” and $B$ “the five flips before it were heads”, it found

Learning $B$ did not change the chance of $A$. Two events like that are independent.

Put $\Pr\{A\mid B\}=\Pr\{A\}$ into the multiplication rule of Conditional probability, $\Pr\{A\cap B\}=\Pr\{B\}\Pr\{A\mid B\}$, and you get the usual definition:

This form has two advantages. It is symmetric, so if $A$ tells nothing about $B$, then $B$ tells nothing about $A$. And it still makes sense when $\Pr\{B\}=0$.

For coin flips, independence is the reason the chance of five heads in a row is $\tfrac12\times\tfrac12\times\tfrac12\times\tfrac12\times\tfrac12=\tfrac1{32}$: each flip’s chance multiplies, untouched by the others.

Independent is not the same as mutually exclusive

Two events that cannot happen together are called mutually exclusive: $\Pr\{A\cap B\}=0$. This is the “disjoint” of What probability measures. It sounds like “having nothing to do with each other”, but it is close to the opposite of independence.

Roll one die. Let $A$ be “it shows 6” and $B$ be “it shows 1”. They are mutually exclusive. Are they independent? Learning that $B$ happened tells you for certain that $A$ did not: $\Pr\{A\mid B\}=0$, not $\tfrac16$. In the product form, $\Pr\{A\cap B\}=0$ but $\Pr\{A\}\Pr\{B\}=\tfrac1{36}$.

Two mutually exclusive events, each with a chance above zero, are never independent: one happening rules the other out.

Independence and two dice

Independence is a fact about numbers, and it does not always look like it from the story. Roll two dice. Is the event “the first die is even” independent of the sum?

Take a sum of 7. Among all 36 rolls, 6 give 7, a chance of $\tfrac16$. Among the 18 rolls whose first die is even, 3 give 7: (2, 5), (4, 3) and (6, 1). That is $\tfrac3{18}=\tfrac16$ again. Learning that the first die is even tells you nothing about whether the sum is 7, so the two events are independent.

Now take a sum of 8. Among all rolls, 5 give 8: a chance of $\tfrac5{36}$. Among the 18 with an even first die, 3 give 8: (2, 6), (4, 4) and (6, 2), a chance of $\tfrac3{18}=\tfrac16$, which is larger. Here the first die being even makes a sum of 8 more likely, so these two events are not independent.

After the clip, drag the sum through every value. Every odd sum, 3, 5, 7, 9 and 11, comes out independent of “first die even”: exactly half of its rolls have an even first die. Every even sum does not. Nothing in the dice story hints at that split; only the counts show it.

More than two events

For three events, it is not enough that each pair is independent. Flip two fair coins. Let $A$ be “the first is heads”, $B$ “the second is heads” and $C$ “the two coins agree”.

Each event has chance $\tfrac12$, and each pair happens together with chance $\tfrac14$: $A\cap B$ is heads-heads, $A\cap C$ is heads-heads, $B\cap C$ is heads-heads. Since $\tfrac12\times\tfrac12=\tfrac14$, every pair is independent.

But all three together are also just heads-heads, with chance $\tfrac14$, not $\tfrac12\times\tfrac12\times\tfrac12=\tfrac18$. Knowing $A$ and $B$ tells you $C$ for certain.

So three events are called mutually independent only when every pair factors and the triple does too:

For more events, every group of two or more must factor in the same way. Separate flips of a coin are mutually independent in this full sense.

Worked example

A machine needs a part that works with chance 0.95 over a year. I assume parts fail independently of each other. Call the chance that part $k$ works its reliability, $r_k$.

1. Three parts in series. The machine needs all three to work, like links in a chain. By independence,

Adding parts in series makes the whole less reliable than any one part.

2. Three parts in parallel. Now any one of three parts is enough, like three spare power supplies. The machine fails only if all three fail, each with chance 0.05:

3. Ten parts in series at 0.99 each: $0.99^{10}=0.904$. Ten good parts make a chain that fails about one year in ten.

4. Spares or a better part? Which fails less often: three parts at 0.95 in parallel, or one part at 0.999? The three: $0.999875>0.999$. Here, three ordinary parts that fail independently beat one excellent part. But only if they really fail independently: three power supplies on the same overloaded cable can fail together.

Try it yourself. A coin has landed heads ten times in a row. The chance of heads on the next flip is still $\tfrac12$.

Where you’ll meet this

Redundant systems. Spare power supplies, the extra engines of an aircraft and disk arrays that store each file on several disks all use the parallel rule. Designers spend most of their effort making sure the copies really fail independently: different power, different cables, different places.

Chains of parts. A circuit board, a supply chain or a long data link works only if every link does: the series rule. Long chains need very reliable links.

Random noise and random numbers. Noise in a signal is often modelled as a fresh, independent random value at every moment, called white noise; Random processes in the signal processing course builds on it. Simulations and security rely on random bits being independent of each other, so that no bit gives away the next.

Long runs. Independence does not mean that a long run corrects itself, as the warning above says. What it does give is The law of large numbers: over many independent flips, the share of heads settles near one half.

Reference card