Priors, posteriors and credible intervals

Three people start with different beliefs about one coin, then watch the same flips. Watch their beliefs move together.

Three priors, the same flips

Before the clip you guessed. After 1000 flips the three people agree to within about a point: their means differ by less than 0.01. In the clip’s run, 703 heads in 1000 flips, they are 0.7026, 0.6980 and 0.7040. At the start they were far apart, one expecting tails, one heads, one with no view at all. The evidence did not erase their priors; it outweighed them.

Each person updates by the rule of Bayesian updating: posterior is prior times likelihood, rescaled. The only difference between them is where they started.

A prior as imaginary flips

A $\mathrm{Beta}(a,b)$ prior has the shape $p^{a-1}(1-p)^{b-1}$. Multiply by the likelihood of $h$ heads in $n$ flips, $p^h(1-p)^{n-h}$, and the powers add:

The posterior is again a beta curve: the data simply add $h$ to the first number and $n-h$ to the second. So the prior acts as if you had already seen $a$ heads and $b$ tails. Its strength, $a+b$, is how many imaginary flips it is worth. The doubtful prior $\mathrm{Beta}(2,8)$ is worth 10 flips with 2 heads; the open-minded $\mathrm{Beta}(1,1)$ is worth only 2.

The posterior mean, $a/(a+b)$ for the new numbers, is a weighted average of the prior’s mean and the share of heads in the data:

The weight $w$ on the prior is its strength over the total. After $n=a+b$ real flips, prior and data count equally. After many more, $w$ is small and every reasonable prior gives nearly the same answer.

The best single guess

The posterior is a whole curve. If you must report one number, which?

Suppose a wrong guess $g$ costs the square of its error. The expected cost is

the same split as bias and variance in Estimators: bias and variance. It is smallest when $g$ is the posterior mean. If instead the cost is the size of the error, the best guess is the posterior median. The peak of the curve is the most believable single value, but under neither cost is it the best report.

Credible intervals

To report a range, cut off 2.5 % of the posterior’s area in each tail. What is left is a 95 % credible interval $[p_\text{lo},p_\text{hi}]$:

Its meaning is direct: given this prior and these data, the belief that $p$ lies in the interval is 0.95. For 7 heads in 10 flips from the flat prior, the posterior is $\mathrm{Beta}(8,4)$ and the interval runs from $0.390$ to $0.891$.

Credible or confidence

The two kinds of interval answer different questions, and they are easy to confuse.

The numbers can still be close. For 7 heads in 10, the flat-prior credible interval is $0.390$ to $0.891$; the Wilson confidence interval is $0.397$ to $0.892$. The plain interval $\hat p\pm1.96\sqrt{\hat p(1-\hat p)/n}$ gives $0.416$ to $0.984$, which leans too far towards 1 for so few flips.

Worked example

1. Seven heads in ten. The open-minded posterior is $\mathrm{Beta}(8,4)$: mean $0.667$, peak $0.7$, median about $0.676$, 95 % credible interval $0.390$ to $0.891$. Its variance is $0.01709$. Guessing the peak, 0.7, costs $0.01709+0.00111=0.01821$ in expected squared error; guessing the mean costs only $0.01709$.

The doubtful person ends at $\mathrm{Beta}(9,11)$, mean $0.45$: exactly halfway between their prior mean, 0.2, and the data’s share, 0.7, because their prior is worth 10 flips and the data are 10 flips ($w=\tfrac12$). The keen person ends at $\mathrm{Beta}(15,5)$, mean $0.75$.

2. More flips, the same share of heads.

Once the flips far outnumber the priors’ strengths, ten times the data shrinks the gap about tenfold, because the weight $w$ on each prior falls like $1/n$.

3. A weak prior. Start from $\mathrm{Beta}(2,2)$ and see 3 heads in 4 flips: the posterior is $\mathrm{Beta}(5,3)$, with mean $5/8=0.625$. The flat prior $\mathrm{Beta}(1,1)$ barely affects any answer once a few dozen flips are in.

Try it yourself. After $n=a+b$ flips, what fraction of the posterior mean comes from the prior? Half. Which cost makes the median the best guess? The size of the error.

Where you’ll meet this

Experiments with a history. A company that has run many similar A/B tests can start each new test from a prior built on the old ones. Small new tests then give sensible answers instead of wild ones.

Medicine with few patients. Trials for rare diseases, where every patient counts, often use priors from earlier studies, stated openly so that others can rerun the analysis with their own.

Rates of rare events. Spam rates, fault rates and click rates are tracked with beta posteriors that update as counts arrive.

Reference card