t-tests

With few data, the mean over its estimated spread does not follow the bell. Watch its curve’s tails shrink as the sample grows.

The t law

In Confidence intervals, putting the sample standard deviation in place of the true one forced a wider edge than 1.96. Here is the curve behind that.

Draw $n$ values from a bell with mean $\mu_0$, and form

with $S$ the sample standard deviation, the random version of $s$. Here $S$ is the spread of the draws, not the sample space of What probability measures. For bell-shaped data, $T$ follows the t law with $n-1$ degrees of freedom, written $\mathrm{t}_{n-1}$. Its curve is a bell with heavier tails. With 2 degrees of freedom, 18.35 % of its area lies beyond $\pm2$; the bell has only 4.55 % there. As the sample grows the t curve closes onto the bell.

The 95 % edge, $t_{0.975,\,n-1}$, falls with the degrees of freedom, from 12.71 at 1 to 1.96 in the limit.

Why the tails are heavier

The bottom of $T$ is random too. When a small sample happens to bunch together, its $S$ is small, and dividing by a small number throws $T$ far out.

Before the clip you guessed. About 18 % of the values land beyond $\pm2$, as the t curve says, not 5 % as the bell would. Of 5000, expect about 918; the clip’s run gave 902.

The one-sample test

To test whether a mean is $\mu_0$, compute $t_\text{obs}=(\bar x-\mu_0)/(s/\sqrt n)$ and read the two-sided $p_\text{val}$ from the t law with $n-1$ degrees of freedom.

The eight response times of Describing data have $\bar x=16.25$ ms and $s=2.9155$ ms. Is their mean 15 ms, the target? The standard error is $1.0308$, so $t_\text{obs}=1.25/1.0308=1.2127$ with 7 degrees of freedom, and $p_\text{val}=0.26$: too little evidence to reject 15.

The 95 % interval of Confidence intervals for the same data, 13.81 to 18.69 ms, contains 15. That is the same statement. A 95 % interval holds exactly the values that a two-sided 5 % test would not reject.

Two samples

The two page layouts of p-values gave old times 10, 11, 12, 16 and new times 14, 15, 17, 19. Their means are 12.25 and 16.25, and their sample variances $6.9167$ and $4.9167$.

Pooled. If both groups share one spread, combine their variances, weighted by their degrees of freedom: $s_\text{p}^2=(3\times6.9167+3\times4.9167)/6=5.9167$. The standard error of the difference is $s_\text{p}\sqrt{1/4+1/4}=1.7200$, so $t_\text{obs}=4.0/1.7200=2.3256$ with $4+4-2=6$ degrees of freedom, and two-sided $p_\text{val}=0.059$.

Welch. If the spreads may differ, use each group’s own variance, $\sqrt{s_1^2/n_1+s_2^2/n_2}$, and a reduced number of degrees of freedom. With equal group sizes the standard error is the same here; the degrees of freedom fall to $5.83$, and $p_\text{val}=0.060$. Welch’s version is the safer default.

The exact shuffle test gave $0.0857$ for the same data. With four values per group, the t law and the shuffle disagree in the second digit: small samples make every method rough.

Pairs

Five people were timed before and after a change: before 20, 24, 18, 22, 26, after 23, 25, 22, 24, 31. Each person was measured twice, so look at each person’s difference: 3, 1, 4, 2, 5. Their mean is 3 and their $s$ is 1.5811, so the standard error is 0.7071 and $t_\text{obs}=4.243$ with 4 degrees of freedom: $p_\text{val}=0.013$.

Treat the same numbers as two unrelated groups and the answer changes. The means are 22 and 25, the variances 10 and 12.5, the pooled variance 11.25, and the standard error $\sqrt{4.5}=2.1213$: $t_\text{obs}=1.414$ with 8 degrees of freedom, $p_\text{val}=0.195$.

The people differ from each other far more than the change moves any one of them. Pairing takes those differences between people away, and the effect shows. A paired test is a one-sample test on the differences.

What the test assumes

• Independence. The values must not influence each other. This matters most and cannot be fixed afterwards.
• A rough bell. For tiny samples the data should be roughly bell-shaped. For larger samples The central limit theorem makes the mean close to a bell anyway, and the test holds up.
• No wild values. One outlier changes both the mean and $s$, as in Describing data; look at the data first.

Worked example

1. The response times against 15 ms. $t_\text{obs}=1.2127$, 7 degrees of freedom, $p_\text{val}=0.26$. At the 5 % level, $1.21$ is below the edge $2.365$: not rejected.

2. The layouts. Pooled: $t_\text{obs}=2.3256$, 6 degrees of freedom, $p_\text{val}=0.059$. Welch: 5.83 degrees of freedom, $p_\text{val}=0.060$.

3. Before and after. Paired: $p_\text{val}=0.013$. Unpaired: $p_\text{val}=0.195$.

4. The 95 % edges.

Try it yourself. For the same $t_\text{obs}$, the $p_\text{val}$ is larger with 3 degrees of freedom than with 30: the tails are heavier. As the sample grows without limit, the t curve becomes the bell.

Where you’ll meet this

Small comparisons. Comparing two versions of a product on a handful of units, or two lab methods on a few samples, is a two-sample t-test.

Measurement against a standard. Checking whether an instrument’s mean reading matches a reference value is a one-sample t-test.

Before and after. Studies that measure the same people, machines or sites twice use paired tests, and gain a great deal from it.

Reference card