Singular value decomposition

The idea

The SVD says every matrix — rectangular, singular, ugly, it does not matter — is secretly three simple operations chained together:

$A = U \Sigma V^\top$

rotate (by $V^\top$ ), stretch along perpendicular axes (by the diagonal $\Sigma$ ), rotate again (by $U$ ). The columns of $V$ are an orthonormal basis of input directions, the columns of $U$ an orthonormal basis of output directions, and the singular values $\sigma_1 \ge \sigma_2 \ge \dots \ge 0$ are gain knobs pairing them up:

$A v_i = \sigma_i u_i$

Feed the matrix its $i$ -th input direction, get the $i$ -th output direction, amplified by $\sigma_i$ . The singular values rank what the matrix actually does: direction 1 is its loudest behavior, and directions with tiny $\sigma_i$ barely matter.

That reading makes the most useful identity in applied linear algebra almost obvious — the matrix as a sum of rank-1 layers:

$A = \sum_{i=1}^{r} \sigma_i \, u_i v_i^\top$

Keep the $k$ loudest layers and you get the truncated SVD $A_k$ , and the Eckart-Young theorem says this greedy choice is optimal: no rank- $k$ matrix is closer to $A$ , and the squared Frobenius error is exactly the discarded energy $\sum_{i > k} \sigma_i^2$ . That single theorem is why SVD sits under PCA, LSA/topic models, image compression, and denoising.

Where does it come from? Multiply out $A^\top A = V \Sigma^2 V^\top$ : the right singular vectors are the eigenvectors of $A^\top A$ , and $\sigma_i = \sqrt{\lambda_i(A^\top A)}$ (all $\lambda_i \ge 0$ because $A^\top A$ is symmetric PSD). Likewise $U$ holds eigenvectors of $A A^\top$ . So the SVD is eigendecomposition applied to the two symmetric matrices any $A$ generates — which is exactly why it exists for every matrix while eigendecomposition of $A$ itself may not.

Worked example

Take

A = \begin{bmatrix} 3 & 0 \\ 4 & 5 \end{bmatrix}.

Step 1 — form $A^\top A$ :

A^\top A = \begin{bmatrix} 25 & 20 \\ 20 & 25 \end{bmatrix}.

Step 2 — find its eigenvalues and singular values:

\begin{aligned} \lambda^2 - 50\lambda + 225 &= 0, \\ \lambda_1 &= 45, & \lambda_2 &= 5, \\ \sigma_1 &= \sqrt{45} = 3\sqrt{5} \approx 6.71, & \sigma_2 &= \sqrt{5} \approx 2.24. \end{aligned}

Two checks account for the determinant and all Frobenius energy:

\begin{aligned} \sigma_1\sigma_2 &= \sqrt{225} = 15 = \det A, \\ \sigma_1^2 + \sigma_2^2 &= 50 = 3^2 + 0^2 + 4^2 + 5^2. \end{aligned}

Step 3 — right singular vectors (eigenvectors of $A^\top A$ ):

v_1 = \frac{1}{\sqrt{2}}(1,1), \qquad v_2 = \frac{1}{\sqrt{2}}(1,-1).

Step 4 — left singular vectors via $u_i = A v_i / \sigma_i$ :

$u_1 = \frac{(3, 9)/\sqrt{2}}{3\sqrt{5}} = \frac{1}{\sqrt{10}}(1, 3), \qquad u_2 = \frac{(3, -1)/\sqrt{2}}{\sqrt{5}} = \frac{1}{\sqrt{10}}(3, -1)$

Check: $u_1 \cdot u_2 = (3 - 3)/10 = 0$ — orthonormal, as promised.

Step 5 — rank-1 truncation. Keeping only the top layer:

$A_1 = \sigma_1 u_1 v_1^\top = \frac{3}{2}\begin{bmatrix} 1 & 1 \\ 3 & 3 \end{bmatrix} = \begin{bmatrix} 1.5 & 1.5 \\ 4.5 & 4.5 \end{bmatrix}$

which captures $\sigma_1^2 / (\sigma_1^2 + \sigma_2^2) = 45/50 = 90\%$ of the squared Frobenius energy. The error matrix is

$A - A_1 = \begin{bmatrix} 1.5 & -1.5 \\ -0.5 & 0.5 \end{bmatrix}$

with squared Frobenius norm $2.25 + 2.25 + 0.25 + 0.25 = 5 = \sigma_2^2$ — Eckart-Young verified on the spot: the residual is exactly the discarded layer $\sigma_2 u_2 v_2^\top$ .

Cheat sheet

Structure

$A = U \Sigma V^\top$ , any $m \times n$ matrix. Thin/economy form keeps $\min(m, n)$ columns; the compact form keeps only the $r = \mathrm{rank}(A)$ with $\sigma_i > 0$ . Action: $A v_i = \sigma_i u_i$ .
$\sigma_i = \sqrt{\lambda_i(A^\top A)}$ ; $V$ = eigenvectors of $A^\top A$ , $U$ = eigenvectors of $A A^\top$ .
Rank-1 layer form: $A = \sum_i \sigma_i u_i v_i^\top$ , layers ordered by importance.

Numbers you can read off the spectrum

Quantity	In terms of singular values
Rank	count of $\sigma_i > 0$ (numerically: above tolerance)
Frobenius norm	$\sqrt{\sum_i \sigma_i^2}$
Spectral norm	$\sigma_1$ (largest amplification factor)
Condition number	$\kappa = \sigma_{\max}/\sigma_{\min}$
Best rank-k error	$\min_{\mathrm{rank}\, B \le k} \lVert A - B \rVert_F^2 = \sum_{i > k} \sigma_i^2$
Energy kept at rank k	$\sum_{i \le k} \sigma_i^2 / \sum_i \sigma_i^2$

Decision rules

Need best low-rank approximation → truncated SVD (Eckart-Young says nothing beats it).
Least squares with a nearly rank-deficient design → pseudoinverse $A^{+} = V \Sigma^{+} U^\top$ (invert only the safely nonzero $\sigma_i$ ).
PCA at scale → SVD of the centered data matrix directly; never form the covariance matrix (squaring the condition number costs precision).
Choosing $k$ → spectral gap or elbow, cumulative energy target (90-95%), or cross-validate a downstream metric.

Gotchas

Singular vector pairs $(u_i, v_i)$ have a joint sign ambiguity — flips across library versions or runs are normal, not bugs.
Recommender-system "SVD" (Netflix Prize) is regularized matrix factorization over observed entries — exact SVD is undefined with missing data, and zero-filling biases factors hard.
Symmetric matrix with negative eigenvalues: $\sigma_i = \lvert \lambda_i \rvert$ , the sign moves into $U$ — singular values are never negative.
Near-equal trailing singular values mean individual trailing vectors are unstable (the subspace is fine, its basis is not).