Chi-square tests

Roll a die 60 times. Each face should come up about 10 times. Watch the gaps between seen and expected add up on a dial.

Squares on the bars

Before the clip you guessed. A fair die sends the needle into the shaded zone about one time in twenty: that is how the zone was drawn. The clip’s 60 rolls gave counts 13, 13, 11, 8, 7 and 8, and the needle stopped at 3.60, far from the zone.

The needle shows the chi-square statistic. For counts $n_k$ in $K$ categories, with $np_k$ the counts expected if the die is fair (here $60\times\tfrac16=10$ each),

Each gap is squared, so a shortfall counts as much as an excess, and scaled by its expected count, so a gap of 4 counts for more where 10 were expected than where 1000 were. A fair die gives small gaps and a small total; a loaded one tends to give a large one.

Where the curve comes from

Each scaled gap $(n_k-np_k)/\sqrt{np_k}$ is roughly a standard bell when the expected count is not too small. A sum of squares of $\mathrm{df}$ independent standard bells follows the chi-square law with $\mathrm{df}$ degrees of freedom, $\chi^2_\text{df}$. It has mean $\mathrm{df}$ and variance $2\,\mathrm{df}$.

The six gaps of the die are not independent: they add up to 0, because the counts add up to 60. That costs one degree of freedom, so the die’s statistic follows $\chi^2_5$, with mean 5 and standard deviation $\sqrt{10}=3.16$. In general a goodness-of-fit test with $K$ categories has $K-1$ degrees of freedom.

The shaded zone starts at the 95 % point of $\chi^2_5$, $11.07$. The 5 % points for 1, 2, 3 and 5 degrees of freedom are $3.841$, $5.991$, $7.815$ and $11.070$. With 2 degrees of freedom the tail has a simple formula, $e^{-x/2}$, so its 5 % point is $2\ln20=5.991$.

Testing a die

Two fair-looking rolls. Counts of 6, 8, 9, 11, 12, 14 have gaps $-4,-2,-1,1,2,4$, whose squares add to 42, so $\chi^2=4.2$: with 5 degrees of freedom, $p_\text{val}=0.52$. Nothing odd. Counts of 4, 6, 8, 9, 12, 21 have squares adding to 182, so $\chi^2=18.2$ and $p_\text{val}=0.003$: the 21 sixes stand out.

A loaded die. After the clip, switch to the loaded die. Its sixes are only a little too common, and 60 rolls catch it about one time in five. With 300 rolls the test catches it about four times in five.

In the clip’s 1000 tests of a fair die, 43 landed past 11.07: 4.3 %, close to the promised 5 %.

With the fair die, about 5 % of repeats land in the shaded zone, give or take 0.7 %. With the loaded die, the share grows with the number of rolls: more data, more power, as in Significance tests.

Tables: is one thing related to another?

Fifty patients get a treatment and 50 do not. Did more of the treated improve?

If improving had nothing to do with the treatment, each cell would hold its row total times its column total over the grand total: here $50\times50/100=25$ in every cell. In general, with $n_{pq}$ the count in row $p$ and column $q$, and $n_{p+}$, $n_{+q}$ the row and column totals, the expected count is $n_{p+}n_{+q}/n$.

Each cell is 5 away from 25, so $\chi^2=4\times25/25=4.0$. A table with $r$ rows and $c$ columns has $(r-1)(c-1)$ degrees of freedom, here 1, and $p_\text{val}=0.0455$.

With one degree of freedom the chi-square statistic is the square of a z statistic. Here the two shares improved are 0.6 and 0.4; their difference over its standard error is $z=0.2/0.1=2$, and $2(1-\Phi(2))=0.0455$, the same test.

Some software applies a continuity correction to $2\times2$ tables, shrinking each gap by 0.5 first: $\chi^2=4\times4.5^2/25=3.24$ and $p_\text{val}=0.0719$. Say which one you used.

When it applies, and what it measures

The bell approximation behind the chi-square curve needs enough counts. A common rule: every expected count at least 5. Rolling a die 6 times, with 1 expected per face, is too few.

Here $\phi$ is a strength coefficient; signal processing pages use the same letter for a phase.

Worked example

1. The die. Counts 6, 8, 9, 11, 12, 14: $\chi^2=4.2$, $p_\text{val}=0.52$. Counts 4, 6, 8, 9, 12, 21: $\chi^2=18.2$, $p_\text{val}=0.003$.

2. A $2\times3$ table. Two groups answer a three-choice question: $(20,30,50)$ and $(30,30,40)$. The column totals are 50, 60 and 90, so the expected rows are both $(25,30,45)$. The six scaled squares are $1,\ 0,\ 0.5556,\ 1,\ 0,\ 0.5556$, so $\chi^2=3.1111$ with $(2-1)(3-1)=2$ degrees of freedom, and $p_\text{val}=e^{-3.1111/2}=0.2110$.

3. Mendel’s peas. Of 556 pea plants, Mendel counted 315, 108, 101 and 32 in four classes that Mendel’s theory put at 9 : 3 : 3 : 1. The expected counts are 312.75, 104.25, 104.25 and 34.75. The scaled squares are $0.016$, $0.135$, $0.101$ and $0.218$, adding to $\chi^2=0.470$ with 3 degrees of freedom: $p_\text{val}=0.925$, a very good fit.

Try it yourself. Counts 5 and 15 against 10 and 10: $\chi^2=(25+25)/10=5$. A $3\times4$ table has $2\times3=6$ degrees of freedom. With 2 degrees of freedom, the chance beyond 5.991 is $e^{-2.9957}=0.05$. Under $H_0$, the die’s statistic averages 5, its degrees of freedom.

Where you’ll meet this

Testing randomness. Random-number generators and hash functions are checked by sorting many outputs into bins and testing whether the counts are even.

Genetics. Ratios of inherited traits, like Mendel’s, are tested against their predicted proportions.

Surveys. Whether an answer depends on age group, region or product is a test of independence on a table of counts.

Model checking. Whether a histogram of data fits a model’s curve is a goodness-of-fit test, with the model’s expected count in each bin.

Reference card