What machine learning is

Could you write a rule that sorts these dots? Below, a line learns one from ten examples. Before it plays: will the line end in the same place if the examples are different?

A rule with three numbers

The line in the picture is a rule: a dot on one side is class A, on the other class B. It has no understanding of dots. It is three numbers, and the clip changed those numbers until the rule agreed with every example.

Each example here is a point with two coordinates, its two measurements or features, $x_1$ and $x_2$. I write them together as $\mathbf{x}=(x_1,x_2)$. Its class, A or B, is its label, $y$. The ten labelled examples are the training set, and I write $m$ for the number of examples, here $m=10$.

A straight line on the plane is every point where a weighted sum is zero. The rule is

The hat on $\hat y$ marks a prediction, the rule’s guess at the label. The numbers $w_1$ and $w_2$ are the weights, $\mathbf{w}=(w_1,w_2)$, and $b$ is the bias term; together they are the rule’s parameters. The sum $w_1x_1+w_2x_2+b$ is the dot’s score: positive on the A side, negative on the B side, zero on the line.

Every choice of the three numbers gives a rule, so the line rules form a family. Learning, here, is picking one member of that family by how well it fits the examples. That is all the word means on these pages: a rule fitted to examples.

How the clip picks it is simple. When a dot is wrong, the line is nudged towards it: for a class A dot, its coordinates are added to the weights and 1 to the bias term; for a class B dot, they are subtracted. The worked example below follows it by hand, and a later lesson studies the rule and when it stops.

Two kinds of question

Predicting a class is one kind of question. Predicting a number is the other.

• Regression predicts a number: a taxi fare from the distance, tomorrow’s temperature from today’s readings. A line rule for one feature is $\hat y=wx+b$, with a plain $w$ for its one weight.
• Classification predicts a class: spam or not, class A or class B.

Both learn from examples that come with their answers, the labels. That is supervised learning. Sometimes there are no answers at all, only the measurements, and the question is whether the examples fall into groups. That is unsupervised learning; k-means clustering finds such groups.

Training and new data

The rule now gets all ten examples right. That was never the goal. The goal is the next dot, one the rule has not seen, and the three grey dots stand for it.

Now the answer to the question at the top. Ten different examples of the same two kinds, the same starting line and the same nudges end on a different line:

Both rules fit their own ten examples perfectly, and they disagree about a new point. So fitting every example does not make a rule right for new ones. How well a rule does on new inputs is what counts, and it is what Bias and variance of a model measures. Nearest neighbours shows a second readout beside the training one, mistakes on new dots, and every lesson that fits something keeps it.

Worked example

The ten examples, in the order they arrive (A first, then taking turns):

• class A: $(2,2)$, $(1,3)$, $(3,1)$, $(3,3)$, $(2,4)$;
• class B: $(-1,-1)$, $(1,-1)$, $(-1,1)$, $(-2,0)$, $(0,-2)$.

The run starts with $\mathbf{w}=(0,1)$ and $b=0$: the rule “A if $x_2>0$”, a level line. A dot is wrong when its score has the wrong sign or is exactly 0. Then, for a class A dot, $\mathbf{w}$ gains $\mathbf{x}$ and $b$ gains 1; for a class B dot, $\mathbf{w}$ loses $\mathbf{x}$ and $b$ loses 1.

In pass 4 the scores are 8, −7, 9, −3, 7, −1, 13, −6, 14, −8 in arrival order: every class A dot positive, every class B dot negative, so nothing moves and the run stops. The rule is

found with four corrections. The dot $(-1,1)$ was corrected twice: one nudge does not always fix the dot it was made for.

New dots. $(3,4)$ scores $6+12-2=16$, class A; $(-2,-2)$ scores $-4-6-2=-12$, class B; $(0,2)$ scores $0+6-2=4$, class A. And $(1,1)$ scores $2+3-2=3$, class A.

Different examples. Class A at $(2,1)$, $(3,3)$, $(1,4)$, $(4,1)$, $(2,3)$ and class B at $(-2,1)$, $(0,-1)$, $(-1,-3)$, $(1,-2)$, $(-3,-1)$, taking turns as before, from the same start. In pass 1, $(-2,1)$ scores 1 and is wrong, giving $\mathbf{w}=(2,0)$, $b=-1$; then $(1,-2)$ scores 1 and is wrong, giving $\mathbf{w}=(1,2)$, $b=-2$. Pass 2 scores 2, −2, 7, −4, 7, −9, 4, −5, 6, −7, all right. The rule is $x_1+2x_2-2>0$, after two corrections. The new dot $(3,-1)$ scores $6-3-2=1$ under the first rule, class A, and $3-2-2=-1$ under the second, class B.

A number rule. With made-up fares, a rule fare $=3+1.5\times$ distance in km predicts $3+1.5\times4=9$ for a 4 km ride. Here the two parameters are the 3 and the 1.5, and fitting would choose them from past rides.

Where you’ll meet this

Spam filters. Each message is turned into numbers from its words, and a rule fitted to mail people have already marked sorts the new mail.

Reading postcodes. A scanned digit is a grid of brightness values; a rule fitted to thousands of labelled digits reads new ones.

Recommendations. What similar people chose before is the training set; the question is what you will choose next.

Machines. Vibration readings from a healthy machine are the examples, and a reading that drifts from them is flagged.

Each is a rule fitted to past examples and judged on new ones.

Reference card