Projections

A point floats above a tilted plane. Which point of the plane is nearest? Watch three ways down, and see which is shortest.

Straight down is not the shortest

Most people expect the nearest point to be straight below, as if the point fell under gravity. The picture says otherwise. Straight down is 3 long. A slanted path to $(2,2,4)$ is shorter, 2.236. The shortest path of all, 1.732 long, lands at $(2,3,5)$, and it is the one that meets the plane at a right angle.

That is the rule for the nearest point. Call it $\mathbf{p}$, and call what is left over $\mathbf{e}=\mathbf{b}-\mathbf{p}$. I write the leftover in bold, $\mathbf{e}$, with no number under it; the unit arrows $\mathbf{e}_1,\mathbf{e}_2$ always carry a number. At the nearest point, $\mathbf{e}$ is at a right angle to the plane.

Why is that the nearest? Take any other point $\mathbf{q}$ of the plane. The step from $\mathbf{p}$ to $\mathbf{q}$ lies in the plane, so it is at a right angle to $\mathbf{e}$. The path from $\mathbf{b}$ to $\mathbf{q}$ is the leftover followed by that step, and by the right-angle rule for lengths,

So every other point is farther. For $\mathbf{q}=(1,2,3)$, the point straight below: $9=3+6$.

After the clip, drag $\mathbf{b}$. The foot follows it, and the right angle never breaks. Drag $\mathbf{b}$ straight along the right-angle path, and the foot does not move at all.

Onto a line

Start with the simplest case, which you met as a shadow in Length and the dot product: project $\mathbf{b}=(1,2)$ onto the line through $\mathbf{a}=(2,1)$.

The foot is some multiple of $\mathbf{a}$, say $\mathbf{p}=c\,\mathbf{a}$. The leftover $\mathbf{b}-c\,\mathbf{a}$ must be at a right angle to $\mathbf{a}$, so its dot product with $\mathbf{a}$ is zero:

Here $\mathbf{a}\cdot\mathbf{b}=2+2=4$ and $\mathbf{a}\cdot\mathbf{a}=5$, so $c=0.8$, $\mathbf{p}=(1.6,0.8)$ and $\mathbf{e}=(-0.6,1.2)$. Check the right angle: $\mathbf{a}\cdot\mathbf{e}=-1.2+1.2=0$. And the lengths: $\lVert\mathbf{p}\rVert^2+\lVert\mathbf{e}\rVert^2=3.2+1.8=5=\lVert\mathbf{b}\rVert^2$.

Onto a plane

Now the plane at the top of the page. It is the column space $C(\mathbf{B})$ of

the plane where the third number is the sum of the first two, from The four fundamental subspaces. Every point of it is $\mathbf{B}\mathbf{x}$ for some weights $\mathbf{x}$, so the foot is $\mathbf{p}=\mathbf{B}\hat{\mathbf{x}}$ for the right weights $\hat{\mathbf{x}}$. The hat marks the best choice.

The leftover must be at a right angle to the whole plane, which means at a right angle to both columns. Two dot products must be zero, and together they are one matrix equation:

These are the normal equations; “normal” is an old word for “at a right angle”. Here $\mathbf{B}^\top\mathbf{B}=\begin{bmatrix}2&1\\1&2\end{bmatrix}$ and $\mathbf{B}^\top\mathbf{b}=(7,8)$. Solving, $\hat{\mathbf{x}}=\tfrac13(2\cdot7-8,\ -7+2\cdot8)=(2,3)$. So $\mathbf{p}=\mathbf{B}\hat{\mathbf{x}}=(2,3,5)$ and $\mathbf{e}=(-1,-1,1)$.

Check: $\mathbf{B}^\top\mathbf{e}=(-1+1,\ -1+1)=\mathbf{0}$. So $\mathbf{e}$ lies in the left null space $N(\mathbf{B}^\top)$, the line of $(1,1,-1)$ on that page. The lengths split as before: $\lVert\mathbf{b}\rVert^2=41=38+3$.

That is the whole picture of a projection. The column space and the left null space meet at a right angle, and every $\mathbf{b}$ splits into one piece in each: $\mathbf{b}=\mathbf{p}+\mathbf{e}$.

The projection matrix

Put the steps together. $\hat{\mathbf{x}}=(\mathbf{B}^\top\mathbf{B})^{-1}\mathbf{B}^\top\mathbf{b}$, and $\mathbf{p}=\mathbf{B}\hat{\mathbf{x}}$, so $\mathbf{p}=\mathbf{P}\mathbf{b}$ with

This $\mathbf{P}$ is the projection matrix. It is not the permutation matrix of Transposes and permutations, which shares the letter. Check it: $\mathbf{P}\mathbf{b}=\tfrac13(6,9,15)=(2,3,5)$.

Three facts about $\mathbf{P}$ follow from the picture.

• Projecting twice changes nothing. A point already on the plane is its own nearest point: $\mathbf{P}(2,3,5)=(2,3,5)$. So $\mathbf{P}^2=\mathbf{P}$.
• It is symmetric. $\mathbf{P}^\top=\mathbf{P}$, as you can read off the matrix above.
• It cannot be undone. Every point on the right-angle path through $\mathbf{b}$ lands on the same foot, so no matrix can send the foot back to the one you started from. $\mathbf{P}$ has no inverse.

The diagonal of $\mathbf{P}$ adds up to $\tfrac13(2+2+2)=2$, the dimension of the plane. For a line, the projection matrix is $\mathbf{a}\mathbf{a}^\top/(\mathbf{a}^\top\mathbf{a})$: for $\mathbf{a}=(2,1)$ it is $\begin{bmatrix}0.8&0.4\\0.4&0.2\end{bmatrix}$, and its diagonal adds up to 1.

Worked example

1. Onto a line. $(1,2)$ onto the line of $(2,1)$: $\mathbf{p}=0.8\,(2,1)=(1.6,0.8)$ and $\mathbf{e}=(-0.6,1.2)$.
2. Onto a plane. $(1,2,6)$ onto the plane of $\mathbf{B}$: $\hat{\mathbf{x}}=(2,3)$, $\mathbf{p}=(2,3,5)$, $\mathbf{e}=(-1,-1,1)$.
3. Already there. If $\mathbf{b}$ is on the plane, $\mathbf{e}=\mathbf{0}$ and $\mathbf{p}=\mathbf{b}$.
4. Twice. $\mathbf{P}\mathbf{P}=\mathbf{P}$.
5. The diagonal. For a projection onto a plane in three dimensions, the diagonal of $\mathbf{P}$ adds up to 2.
6. Predict. Drag $\mathbf{b}$ along the right-angle path through it. The foot stays where it is.

Where you’ll meet this

A mains hum has a known shape. Suppose the shape is $\mathbf{u}=(1,0,-1,0)$ and a recording is $\mathbf{x}=(4,1,0,1)$. The hum’s share is $(\mathbf{u}\cdot\mathbf{x})/(\mathbf{u}\cdot\mathbf{u})=4/2=2$, and removing it leaves $\mathbf{x}-2\mathbf{u}=(2,1,2,1)$, which is at a right angle to $\mathbf{u}$.

Removing the mean of a list is a projection too. Project $(2,4,6)$ onto $(1,1,1)$: the foot is $(4,4,4)$, the mean repeated, and the leftover $(-2,0,2)$ is the list with its mean removed. The shadow of a game character under an overhead light is a projection onto the floor, with the matrix that keeps the first two numbers and sets the height to 0.

Reference card