Logistic regression

A score says which side of a line a dot is on, not how sure to be. Below, scores become chances. Before it plays: where will the boundary run?

Squashing a score

A line rule, as in What machine learning is, gives each point a score,

positive on one side of the line and negative on the other. Here $z$ is only the score; it is not the variable of a z-transform or a standardised value. A score can be any number, so it is not a chance. Turn it into one with the sigmoid,

an S-shaped curve that runs from 0 far to the left to 1 far to the right, through one half at $z=0$. I always write it with its argument, $\sigma(z)$, because a bare $\sigma$ means a spread elsewhere. The model’s chance of class A is

This is logistic regression: a linear score, squashed into a probability. Despite its name it predicts a class, through the chance of that class, not a number on a scale. The curve is symmetric: $\sigma(-z)=1-\sigma(z)$, so the chance of class B is $\sigma(-z)$.

The score is the log-odds

Undo the sigmoid. From $\hat p=1/(1+e^{-z})$, the ratio of the chance of class A to the chance against it is

That ratio is the odds, and the score is the logarithm of the odds, the log-odds or logit. Each unit added to the score multiplies the odds by $e\approx2.718$.

The boundary is a line

To turn a chance into a class, compare it with a threshold. At the usual threshold, one half, the rule is class A when $\hat p>0.5$. Since $\sigma(z)>0.5$ exactly when $z>0$, the decision boundary is

a straight line, whatever the S-curve looks like. The curve bends the chances, not the boundary.

Now the answer to the question. The boundary runs straight between the crescents, rising gently. A straight line cannot follow the curved gap between them, so the dots at the tips of each crescent, where they reach into the other, stay on the wrong side; those are the ringed dots, 13 of the 100. Decision trees and Multilayer networks draw boundaries that can bend.

Steepness

Multiply the weights and the bias term together by a positive number $a$, the steepness the slider sets. Every score is multiplied by $a$, so the points where $z=0$ do not move: the boundary stays exactly where it is. What changes is how fast the chance rises across it. With $a=4$ a point a little way from the line already gets a chance near 0 or 1, and the uncertain band is narrow. With $a=0.25$ the band is wide, and even points far from the line get modest chances.

So the length of the weight vector says how sure the model is, and the direction of $\mathbf{w}$ and $b$ together says where the boundary lies. However steep, the line still cannot bend round the tips of the crescents.

Choosing the weights

The weights in the clip were fitted by Gradient descent, 200 steps with learning rate 0.5 from all weights zero, on a loss that punishes confident mistakes: a dot of class A given a chance near 0 costs a great deal. Cross-entropy loss builds that loss and its gradient. Because the crescents overlap, no line separates them, and the fitted weights stay finite. Two hundred steps bring them close to the bottom of that loss, not onto it: run to the end, the second weight would be 2.473 instead of 2.437, a line a little steeper in its shading and almost exactly where it is.

What 0.9 means

A chance of 0.9 does not mean the model is right 90 % of the time on that one dot. One dot is either class A or not, and one outcome cannot prove a chance right or wrong. A chance is a statement about many cases: among all the dots the model gives about 0.9, about 9 in 10 should be class A. A model that keeps to that is well calibrated. Whether it does is something to check on new data.

The bars show how far a straight-line model’s chances can be taken as rates on these crescents: the closer each bar to the diagonal, the more its chances mean what they say. At the top they nearly do. Between 0.6 and 0.8 they overstate: the model says about 7 in 10 and the dots say 6. A line that cannot follow the crescents’ curve cannot get every rate right either.

Thresholds

One half is a choice, not a law. If missing a faulty part costs far more than a needless inspection, flag a part when its chance of a fault passes 0.1. A lower threshold catches more of the class and wrongly flags more of the rest. Measuring a classifier counts both kinds of mistake.

Worked example

One feature. A component’s chance of a fault from its temperature, with weight $w=0.5$ per °C and $b=-30$. The score is $z=0.5\times\text{temperature}-30$, zero at 60 °C: the boundary.

• At 64 °C, $z=32-30=2$ and $\hat p=1/(1+e^{-2})=0.8808$. The odds are $e^2=7.39$ to 1.
• At 56 °C, $z=-2$ and $\hat p=0.1192$, which is $1-0.8808$.
• At 70 °C, $z=5$ and $\hat p=0.9933$.
• Each extra degree adds 0.5 to the score, so it multiplies the odds by $e^{0.5}=1.649$.

Two features. Take $\mathbf{w}=(2,-1)$ and $b=1$. The boundary is $2x_1-x_2+1=0$, the line $x_2=2x_1+1$. The point $(1,1)$ scores $2-1+1=2$, so $\hat p=0.8808$; the point $(0,3)$ scores $-3+1=-2$, so $\hat p=0.1192$.

Steeper. Multiply everything by 3: $\mathbf{w}=(6,-3)$, $b=3$. Now $(1,1)$ scores 6 and $\hat p=0.9975$, but the boundary, $6x_1-3x_2+3=0$, is the same line $x_2=2x_1+1$.

Backwards. At what temperature is $\hat p=0.1192$? At $z=-2$, so $0.5\times\text{temperature}=28$: 56 °C. And $\sigma(0)=0.5$.

Where you’ll meet this

Medical risk scores. Measurements such as age and blood pressure are weighted, summed and turned into a chance of illness within some years.

Credit scoring. The chance that a loan is repaid, from the applicant’s records.

Clicks. The chance that a person clicks a link, from what is known about the link and the page.

Machine failure. The chance that a machine fails before its next service, from its temperature and vibration.

Reference card