Principal components

Which way does a cloud of points spread the most? Watch a line turn through the cloud while the spread of the points’ shadows is traced.

The spread along a direction

The six points in the picture are already centred: their average is the origin. That is the first step of every analysis here, since spread is measured from the middle.

Pick a direction, a unit vector $\mathbf{w}$. Each point $\mathbf{y}$ casts a shadow on the line of $\mathbf{w}$ of signed length $\mathbf{y}\cdot\mathbf{w}$, as on Length and the dot product. The spread of those shadows is their variance. Here and below, the variance divides the sum of squares by $n-1=5$, one less than the number of points, as in Covariance and correlation.

Along the horizontal axis the shadows are the $x$ values, with variance $76/5=15.2$; along the vertical axis the $y$ values, with variance $34/5=6.8$. Neither is the largest. The line in the picture finds 18, along $(2,1)$.

The covariance matrix

Stack the centred points as the rows of a data matrix $\mathbf{Y}$, one row per point. The covariance matrix is

Its diagonal holds the variance of each coordinate, and its off-diagonal entry the covariance of the two. For the six points, $\sum x^2=76$, $\sum y^2=34$ and $\sum xy=28$, so

The variance of the shadows along a unit $\mathbf{w}$ is the energy of Positive definite matrices:

$\mathbf{S}$ is symmetric, so by Symmetric matrices that is $\lambda_1u_1^2+\lambda_2u_2^2$ in its eigenvector coordinates. The largest it can be, for a unit $\mathbf{w}$, is the top eigenvalue $\lambda_1$, reached along the top eigenvector. That direction is the first principal component. The next one, perpendicular to it, is the second.

Here $\mathrm{tr}\,\mathbf{S}=22$ and $\det\mathbf{S}=103.36-31.36=72$, so the eigenvalues are 18 and 4. The eigenvectors are $(2,1)$, since $\mathbf{S}(2,1)=(36,18)=18\,(2,1)$, and $(-1,2)$, since $\mathbf{S}(-1,2)=(-4,8)=4\,(-1,2)$. The two components carry $18/22=81.8\,\%$ and $4/22=18.2\,\%$ of the total variance.

By the SVD

You don’t need to form $\mathbf{S}$. Take the SVD of the data matrix itself, $\mathbf{Y}=\mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^\top$, from The singular value decomposition. Then $\mathbf{Y}^\top\mathbf{Y}=\mathbf{V}\boldsymbol{\Sigma}^\top\boldsymbol{\Sigma}\mathbf{V}^\top$, so:

• the columns of $\mathbf{V}$ are the principal directions;
• the eigenvalues of $\mathbf{S}$ are $\lambda_k=\sigma_k^2/(n-1)$;
• each point’s coordinate along direction $k$, its score, is entry $k$ of $\mathbf{Y}\mathbf{v}_k$.

For the six points, $\sigma_1=\sqrt{90}=9.487$ and $\sigma_2=\sqrt{20}=4.472$, and $90/5=18$, $20/5=4$. Along $(2,1)/\sqrt5$, the point $(3,4)$ scores $10/\sqrt5=4.472$, $(5,0)$ also 4.472, and $(2,1)$ scores 2.236. Keep only the first component and each point is replaced by its shadow; the squared error left over is $\sigma_2^2=20$, as in Low-rank approximation.

Programs use the SVD route: forming $\mathbf{Y}^\top\mathbf{Y}$ squares the sensitivity to rounding, as Conditioning explains.

Two traps

The method finds the direction of most spread. It can’t find anything else, and the next picture shows two ways that matters.

Curves are invisible. The eight points on the ring have $\sum x^2=\sum y^2=4$ and $\sum xy=0$, so $\mathbf{S}=\tfrac47\mathbf{I}$: no direction stands out. The covariance only sees spread, not shape.

Units decide. Measure $x$ in millimetres instead of centimetres, multiplying it by 10, and the top direction swings to 2.12°, almost along $x$, with 99.7 % of the variance. Nothing about the data changed except the units. The usual fix is to standardise first: divide each coordinate by its own spread. For the six points the standardised covariance is the correlation matrix, with correlation $\hat\rho_{xy}=28/\sqrt{76\times34}=0.5508$, and its eigenvalues are $1+\hat\rho_{xy}=1.5508$ and $1-\hat\rho_{xy}=0.4492$, along $(1,1)$ and $(1,-1)$.

Worked example

1. The covariance route. Sums 76, 34 and 28; $\mathbf{S}=\begin{bmatrix}15.2&5.6\\5.6&6.8\end{bmatrix}$; eigenvalues 18 and 4 along $(2,1)$ and $(-1,2)$; shares 81.8 % and 18.2 %.

2. The SVD route. $\sigma=9.487$ and 4.472, and $\sigma^2/5=18$ and 4. The squared size of $\mathbf{Y}$ is $110=90+20$.

3. Scores. On the first component: 4.472, 4.472 and 2.236 for $(3,4)$, $(5,0)$ and $(2,1)$, and the negatives for the other three.

4. Standardised. $\hat\rho_{xy}=0.5508$, eigenvalues 1.5508 and 0.4492.

Try these before you read the answers.

• The variance along $(2,1)/\sqrt5$? 18.
• If every point lies on one line through the origin, what is the second eigenvalue? 0.
• Measure $x$ in millimetres instead of centimetres. Does the top direction change? Yes: it swings towards $x$. Standardise first.

Where you’ll meet this

Cleaning brain recordings. An eye blink shows up on many electrodes at once, in one fixed mix. In the many-channel recording it is one strong principal direction, which can be found and removed.

Faces and genes. Pictures of faces, and the genetic data of many people, have a few principal directions that carry most of their variation; maps of human populations have been drawn from the first two.

Finance. A few principal components describe how a whole set of interest rates moves: up together, steeper or flatter.

In every case the first component is the direction of most spread, not of most meaning. The meaning is something the analyst has to supply.

Reference card