The singular value decomposition

Every matrix turns a circle into an ellipse. Watch it happen in three moves: a turn, a stretch, and another turn.

A circle becomes an ellipse

Take the matrix $\mathbf{A}=\begin{bmatrix}3&0\\4&5\end{bmatrix}$. As on A matrix moves the plane, it moves every point of the plane, and the unit circle lands on some curve. For every matrix that curve is an ellipse, or a segment if the matrix flattens the plane.

The picture at the top builds that ellipse in three moves. A turn by −45° (that is, $-\pi/4\approx-0.7854$ rad) brings two special perpendicular arrows onto the axes. A stretch along the axes pulls the circle into an upright ellipse. A second turn, by 71.57°, swings the ellipse into place. Turn, stretch, turn, with a flip in one of the turns when the matrix flips the plane: that is all any matrix does to the plane.

Name the pieces

The two arrows that land on the axes are the right singular vectors, $\mathbf{v}_1$ and $\mathbf{v}_2$. The stretch factors are the singular values, $\sigma_1\ge\sigma_2\ge0$. On this page $\sigma$ is a stretch factor, not a standard deviation or the real part of $s$. Where each arrow ends up, as a unit vector, is a left singular vector, $\mathbf{u}_1$ or $\mathbf{u}_2$. Together:

Put the $\mathbf{v}_k$ as the columns of $\mathbf{V}$, the $\mathbf{u}_k$ as the columns of $\mathbf{U}$, and the $\sigma_k$ on the diagonal of $\boldsymbol{\Sigma}$. Both $\mathbf{U}$ and $\mathbf{V}$ have orthonormal columns, as in Orthonormal bases and Gram–Schmidt, so their inverses are their transposes. Then

Read it right to left: $\mathbf{V}^\top$ turns, $\boldsymbol{\Sigma}$ stretches, $\mathbf{U}$ turns. When the matrix flips the plane, one of the two turns also flips. This is the singular value decomposition, or SVD. These $\mathbf{U}$ and $\mathbf{V}$ play the same part as the unitary $\mathbf{U}$ of the Fourier matrix page and the basis $\mathbf{V}$ of Independence, basis and dimension: sets of perpendicular directions.

Where the singular values come from

Multiply $\mathbf{A}$ by its own transpose. With $\mathbf{A}=\mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^\top$ and $\mathbf{U}^\top\mathbf{U}=\mathbf{I}$:

That is the spectral form of Symmetric matrices. So the $\mathbf{v}_k$ are the eigenvectors of the symmetric matrix $\mathbf{A}^\top\mathbf{A}$, and its eigenvalues are the squares $\sigma_k^2$. Once you have $\mathbf{v}_k$ and $\sigma_k$, the left vectors follow from $\mathbf{u}_k=\mathbf{A}\mathbf{v}_k/\sigma_k$.

For our matrix:

Its eigenvalues are 45, along $(1,1)$, and 5, along $(1,-1)$. So $\sigma_1=\sqrt{45}=6.708$ and $\sigma_2=\sqrt5=2.236$. Then $\mathbf{A}\mathbf{v}_1=(3,9)/\sqrt2$, of length $\sqrt{45}$, gives $\mathbf{u}_1=(1,3)/\sqrt{10}$, at 71.57°. With $\mathbf{v}_2=(-1,1)/\sqrt2$, $\mathbf{A}\mathbf{v}_2=(-3,1)/\sqrt2$ gives $\mathbf{u}_2=(-3,1)/\sqrt{10}$. In matrices:

Both are turns. Each singular vector could also be flipped, together with its partner; the choice of signs is a convention.

Two checks need no vectors. The largest stretch of any unit arrow is $\sigma_1$, so $\lVert\mathbf{A}\mathbf{x}\rVert\le\sigma_1\lVert\mathbf{x}\rVert$ for every $\mathbf{x}$. And the squares of all the entries add up to the squares of the singular values. That sum is the Frobenius norm squared, $\lVert\mathbf{A}\rVert_\mathrm{F}^2$: here $9+0+16+25=50=45+5$.

Rank and the four subspaces

Now a matrix that flattens the plane:

Its columns are equal, so every output lies on the line of $(1,2)$. Here $\sigma_1=\sqrt{10}=3.162$ and $\sigma_2=0$: the circle collapses to a segment. The singular vectors hand you all four subspaces of The four fundamental subspaces at once:

• $\mathbf{u}_1=(1,2)/\sqrt5$ spans the column space;
• $\mathbf{u}_2=(2,-1)/\sqrt5$ spans the left null space;
• $\mathbf{v}_1=(1,1)/\sqrt2$ spans the row space;
• $\mathbf{v}_2=(1,-1)/\sqrt2$ spans the null space, the direction stretched by 0.

The rank is the number of singular values that are not zero.

A sum of rank-one pieces

Multiply $\mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^\top$ out by columns times rows, as in Matrix multiplication:

Each piece is a column times a row, a matrix of rank one. For our $\mathbf{A}$ the pieces are

The SVD works for any shape. The $3\times2$ matrix $\mathbf{B}=\begin{bmatrix}1&0\\0&1\\1&1\end{bmatrix}$ of A matrix moves the plane has $\mathbf{B}^\top\mathbf{B}=\begin{bmatrix}2&1\\1&2\end{bmatrix}$, with eigenvalues 3 and 1. So $\sigma_1=\sqrt3=1.732$ and $\sigma_2=1$, with $\mathbf{u}_1=(1,1,2)/\sqrt6$ and $\mathbf{u}_2=(1,-1,0)/\sqrt2$. It turns the flat circle into an ellipse tilted in space.

Worked example

1. The full decomposition. For $\mathbf{A}=\begin{bmatrix}3&0\\4&5\end{bmatrix}$: $\mathbf{A}^\top\mathbf{A}$ has eigenvalues 45 and 5, so $\sigma=6.708$ and 2.236. $\mathbf{A}\mathbf{A}^\top=\begin{bmatrix}9&12\\12&41\end{bmatrix}$ has the same two eigenvalues (trace 50, determinant 225), with eigenvectors $(1,3)$ and $(-3,1)$: the left singular vectors. The pieces rebuild $\mathbf{A}$, $\lVert\mathbf{A}\rVert_\mathrm{F}^2=50=45+5$, and $\sigma_1\sigma_2=15=\det\mathbf{A}$.

2. Rank one. $\begin{bmatrix}1&1\\2&2\end{bmatrix}=(1,2)\,(1,1)^\top$, a column times a row: $\sigma_1=\sqrt5\cdot\sqrt2=\sqrt{10}$, $\sigma_2=0$, rank 1.

3. A tall matrix. $\mathbf{B}$ has $\sigma=\sqrt3$ and 1.

4. The Fourier matrix. Every singular value of $\mathbf{F}$ for $N=4$ is 2.

Try these before you read the answers.

• Singular values of $3\mathbf{I}$? 3 and 3: the circle stays a circle.
• The longest $\mathbf{A}\mathbf{x}$ for a unit $\mathbf{x}$, for the worked $\mathbf{A}$? 6.708.
• A symmetric matrix has eigenvalues 3 and $-2$. Its singular values? 3 and 2: the sign moves into $\mathbf{U}$ or $\mathbf{V}$, which then flips.

Where you’ll meet this

The engine of later pages. Keeping only the large pieces compresses a matrix, in Low-rank approximation. The SVD of a data table finds its main directions, in Principal components. Least-squares routines in software solve through it, in The pseudo-inverse, and the ratio $\sigma_1/\sigma_2$ measures how sensitive a system is, in Conditioning.

Recommendations and text. A table of ratings, people by films, is factored by its SVD; the few largest pieces capture tastes. The same is done to tables of words by documents to find topics.

Antenna and microphone arrays. The SVD of an array’s recorded data splits it into a few strong directions, the signals, and many weak ones, the noise.

Reference card