Describing data

Nine response times sit on a line. Drag the slowest one far out, and watch the mean and the median.

A sample and its summaries

The nine times are a sample: a few measurements taken from something much larger, here every request the server will ever answer. That larger whole is the population. A number computed from the sample, such as its average, is a statistic. It describes these nine values; the population’s own average is a different number, which the sample can only estimate.

I write the sample values $x_1,\dots,x_n$, here $n=9$. Sorted from smallest to largest they are $x_{(1)}\le x_{(2)}\le\dots\le x_{(n)}$, with the index in brackets.

Two centres

The sample mean is the plain average,

The median is the middle value once the data are sorted: $x_{(5)}$ of nine values. With an even number of values it is halfway between the two middle ones.

Before the clip you guessed what the median would do. It stayed at 16. As the ninth value ran out to 95, the mean followed it from 16.78 to 25, because every millisecond added to one value adds $1/9$ ms to the mean. The median only asks which value is in the middle, and the middle value never changed.

After the clip, drag the ninth value yourself. The slider starts at 21, already above the median, and runs right to 120. Dragging right, the mean climbs from 16.78 to 27.78, while the median stays at 16 the whole way. Only a ninth value below 16, which the slider does not reach, could move the median.

At 95, the mean of 25 is larger than eight of the nine values. It is not a typical response time at all.

Spread: three ways

The sample standard deviation. As in Variance and standard deviation, spread is measured by squared distances from the centre, but for a sample the sum is divided by $n-1$, not $n$:

Why $n-1$ is the subject of Estimators: bias and variance, which settles it with an exact count. Here the letter $s$ is the sample standard deviation; signal processing pages use $s$ for something else.

Quartiles and the IQR. The quartiles $Q_1$ and $Q_3$ cut off the lowest quarter and the highest quarter of the sorted data. Different programs place them slightly differently; I use NumPy’s default: the value a fraction $f$ of the way along is at position $(n-1)f$, counting the sorted values from 0, with a straight line between neighbours. The interquartile range, $\mathrm{IQR}=Q_3-Q_1$, is the width of the middle half.

The MAD. The median absolute deviation is the median of the distances from the median. Multiplied by 1.4826 it matches the standard deviation for bell-shaped data, but it ignores a few wild values.

The box plot

A box plot draws these summaries on one line. The box runs from $Q_1$ to $Q_3$ with a line at the median. Two fences sit $1.5\,\mathrm{IQR}$ beyond the box, at $Q_1-1.5\,\mathrm{IQR}$ and $Q_3+1.5\,\mathrm{IQR}$. Whiskers reach out to the furthest data inside the fences, and any point beyond a fence is drawn on its own as an outlier.

The shape: histograms and smooth curves

A summary is only as good as the shape behind it, so look at the shape first. Here are 200 waiting times of the lopsided kind in Continuous random variables.

The smooth curve is a kernel density estimate: put a small bell on every data point and add them up, scaled so the area is 1. The width of each bell plays the part of the bin width. Too narrow and every point shows as its own spike; too wide and the shape disappears. A common rule of thumb, Silverman’s, gives a width of about 0.26 for 200 values of this shape.

The long right tail shows in the summaries too: the mean sits above the median. For this waiting-time law the exact values are 1 and $\ln2=0.6931$. The 200 draws in the picture give 1.0435 and 0.6634.

Worked example

1. The nine times, with the outlier. The values add to 225, so $\bar x=25$. The median is 16. By NumPy’s rule $Q_1=15$ and $Q_3=19$, so $\mathrm{IQR}=4$ and the fences are $15-6=9$ and $19+6=25$: the value 95 is an outlier. The squared deviations from 25 add to 5572, so $s^2=5572/8=696.5$ and $s=26.39$ ms.

2. The eight times, without it. The values add to 130, so $\bar x=16.25$, and the median is $15.5$. The quartiles are $14.75$ and $18.25$. The squared deviations add to $59.5$, so $s^2=59.5/7=8.5$ and $s=2.9155$ ms.

One point multiplied the standard deviation by about nine: $26.39/2.9155=9.05$.

3. The MAD of the nine. The distances from the median 16, sorted, are $0,1,1,2,2,3,4,5,79$, and their median is 2. Scaled, $1.4826\times2=2.97$ ms: close to the clean eight’s $s$, as if the outlier were not there.

4. Why the divisor matters. Two values, 10 and 14, have mean 12 and squared deviations adding to 8. Dividing by $n=2$ gives 4; dividing by $n-1=1$ gives 8. Estimators: bias and variance shows which is right on average.

5. A median filter. A sensor reports 4, 50, 5, where 50 is a glitch. The mean of the three is $19.67$; the median is 5. Replacing each sample by the median of its neighbours removes such spikes.

Try it yourself. The median of $3,9,1,7$ is 5, halfway between 3 and 7. The values $1,2,3,100$ have mean $26.5$ and median $2.5$. Adding 10 to every value moves the mean and median by 10 and leaves $s$ unchanged.

Where you’ll meet this

Response times. Engineers often report the median and the 95th percentile of response times, not the mean: a few very slow requests would drag the mean and hide what most users see.

Incomes and prices. Incomes and house prices have long right tails, so official figures usually quote the median. The mean is pulled up by the few largest values.

Quality reports. Box plots put many batches side by side on one page, each with its centre, its middle half and its outliers.

Reference card