Maximum likelihood

Three data points sit on a line. A bell slides along them, and the heights it gives the points are multiplied. Watch where the product peaks.

Slide the model, multiply the heights

I have data, and a model with a knob: here a bell of known width whose centre $\mu$ I do not know. Each setting of the knob gives each data point a height. The idea of maximum likelihood is to choose the setting that makes the data I saw as probable as possible.

The heights multiply because the three points were drawn independently: the chance of seeing all three is a product. That product, as a function of the knob, is the likelihood:

where $f(x;\theta)$ is the model’s density (or its pmf, for counts) at $x$ when the knob is set to $\theta$. The best setting, $\hat\theta$, is the maximum likelihood estimate.

Before the clip you guessed where the peak would be. It is at $\mu=5$, the middle value, which is also the mean of the three. At $\mu=3$ the bell sits on the left point and gives the right point, 7, a low height; the product suffers. At $\mu=5$ no point is far from the centre.

Take the logarithm

Products of many small numbers are awkward: with 1000 data points the likelihood is smaller than any computer can store. Take the logarithm instead. It turns the product into a sum, and since the logarithm only ever rises, the peak stays at the same $\theta$:

This is the log-likelihood. For the bell of width 2,

so for the three points the log-likelihood is $-\tfrac38(\mu-5)^2$ plus a constant: a parabola with its top at 5, the curve under the likelihood in the picture.

Three recipes

To find the peak, set the slope of $\ell$ to zero.

A coin. $h$ heads in $n$ independent flips with chance $p$ of heads: $\ell(p)=h\ln p+(n-h)\ln(1-p)$. Its slope is $h/p-(n-h)/(1-p)$, which is zero at

A bell. For bell-shaped data with unknown centre and width, the slopes give $\hat\mu=\bar x$, the sample mean, and

with divisor $n$, so biased low, as Estimators: bias and variance showed. Maximum likelihood is not always unbiased.

Waiting times. For gaps $t_1,\dots,t_n$ from the exponential law of rate $\lambda$, $\ell(\lambda)=n\ln\lambda-\lambda\sum_kt_k$. Its slope $n/\lambda-\sum_kt_k$ is zero at

one over the mean gap.

A coin, and how sure to be

The likelihood also shows how sharply the data pin the parameter down.

The peak is at $\hat p=0.7$ in both cases. With 10 flips the curve is wide, so many values of $p$ fit nearly as well. With 100 flips it is narrow. Its width is measured by the standard error of $\hat p$,

$0.1449$ for 10 flips and $0.0458$ for 100: ten times the data, a third of the width.

Two properties make maximum likelihood a default way to fit a model. Under mild conditions on the model, as the sample grows, $\hat\theta$ closes in on the true value: it is consistent. And for large samples its distribution over repeated samples is close to a bell centred on the truth, with a width that shrinks like $1/\sqrt n$. I state these without proof.

Worked example

1. Seven heads in ten. At the estimate,

A fair coin gives $L(0.5)=0.5^{10}=0.000977$. So $\hat p=0.7$ makes the data $2.28$ times as probable as a fair coin does: more probable, but not overwhelmingly.

2. The bell, with its width unknown too. For the data 3, 5 and 7, $\hat\mu=5$. The squared deviations add to 8, so $\hat\sigma^2=8/3=2.667$ and $\hat\sigma=1.633$. The sample variance of Describing data divides by 2 instead and gives $s^2=4$.

3. Waiting times. Four gaps between clicks, $0.2,\ 0.9,\ 0.4,\ 0.5$ s, add to 2.0 s. So $\hat\lambda=4/2=2$ per second, the rate of the Geiger counter in Random arrivals. The log-likelihood $\ell(\lambda)=4\ln\lambda-2\lambda$ peaks there at $4\ln2-4=-1.2274$.

4. Noise that is bell-shaped. If each measurement is a model value plus bell-shaped noise of fixed width, then $-\ell$ is, apart from a constant, $\sum_k(x_k-\mu_k)^2/(2\sigma^2)$. Making the likelihood largest is making the sum of squared errors smallest: maximum likelihood is least squares.

Try it yourself. Three heads in 12 flips give $\hat p=3/12=0.25$. A mean gap of 0.25 s gives $\hat\lambda=4$ per second. If every data value is shifted by 10, the estimate of the bell’s centre shifts by 10.

Where you’ll meet this

Fitting models. Rates of events, failure times, the gain of a channel, the frequency of a tone in noise: many fitted parameters in science and engineering are maximum likelihood estimates, or close to them.

Classification. Logistic regression, the standard model for yes-or-no outcomes, is fitted by maximum likelihood.

Searching for a peak. Many estimates are found the way the picture found $\mu$: slide a template along, score each position, keep the best. Pitch trackers slide over candidate frequencies; radar slides over delays.

Reference card