Principal component analysis

The idea

A centered cloud of points in $d$ dimensions usually does not fill all $d$ dimensions equally — it is elongated some ways, thin others. PCA finds the cloud's natural axes: the first principal component is the direction of maximal variance, the second is the most-variance direction perpendicular to the first, and so on. Keeping the top $k$ axes and dropping the rest is the best linear compression of the data in a sense made precise below.

The objective, and why eigenvectors appear. Project centered data onto a unit direction $w$; the projected variance is $w^\top \Sigma w$ where $\Sigma$ is the covariance matrix. Maximize it subject to $\lVert w \rVert = 1$ with a Lagrange multiplier:

$\max_{w} \; w^\top \Sigma w - \lambda (w^\top w - 1) \quad \Longrightarrow \quad \Sigma w = \lambda w$

The stationary points are eigenvectors of $\Sigma$, and the variance achieved at an eigenvector is its eigenvalue ($w^\top \Sigma w = \lambda$). So the top eigenvector wins, the second component is the next eigenvector, and orthogonality is not an extra assumption — it is delivered free by the spectral theorem, since $\Sigma$ is symmetric.

Why "maximize variance" is not arbitrary. Total variance $\sum_i \lambda_i$ is fixed by the data. Each point splits, by Pythagoras, into its projection onto the kept subspace plus a perpendicular residual:

$\underbrace{\textstyle\sum_{i \le k} \lambda_i}_{\text{variance kept}} \; + \; \underbrace{\textstyle\sum_{i > k} \lambda_i}_{\text{mean squared reconstruction error}} \; = \; \text{total variance}$

Maximizing kept variance and minimizing squared reconstruction error are the same optimization. PCA is exactly the $k$-dimensional linear view that loses the least of the data — which is also why it coincides with the truncated SVD of the centered data matrix.

The recipe (and where each classic mistake hides):

1. Center features with training-set means (without centering, PCA decomposes the second moment and the leading component may be dominated by the mean offset rather than variation around the mean).
2. Standardize if features have incomparable units (skip → big-unit features hijack the components).
3. Decompose: SVD of $X_c$ in practice, $X_c = U S V^\top$. Directions = columns of $V$; variances $\lambda_i = s_i^2/(n-1)$; scores = $US$.
4. Pick $k$ from the explained-variance ratio $\lambda_i / \sum_j \lambda_j$, an elbow, or downstream cross-validation.
5. Transform new data with the same means, scales, and $V$ — refitting on test data is leakage.

See it happen in three dimensions — the plane below is fit live from the cloud's own covariance matrix, then every point collapses onto it:

Worked example

Four points, already centered (mean is the origin): $(3,3), \; (-3,-3), \; (1,-1), \; (-1,1)$.

Covariance (divisor $n - 1 = 3$):

$\Sigma = \frac{1}{3}\begin{bmatrix} 20 & 16 \\ 16 & 20 \end{bmatrix}, \qquad \text{e.g. } \Sigma_{11} = \tfrac{9 + 9 + 1 + 1}{3} = \tfrac{20}{3}$

Eigenpairs. By symmetry the eigenvectors are the diagonals:

The explained-variance ratio of PC1 is

Scores on PC1 ($z=x\cdot w_1$):

The score variance checks the eigenvalue directly:

Reconstruction with $k = 1$. Rebuild each point as $z \, w_1$: $(3,3)$ and $(-3,-3)$ lie exactly on the PC1 line, so they reconstruct perfectly. $(1,-1)$ and $(-1,1)$ are perpendicular to PC1, so both collapse to the origin, each with squared error $2$. Average the squared errors with the same $n - 1$ divisor as the covariance: $(0 + 0 + 2 + 2)/3 = 4/3 = \lambda_2$ — exactly the discarded eigenvalue, confirming kept variance ($12$) plus reconstruction error ($4/3$) equals total variance ($40/3$).

The scaling trap in one line. Rescale the second feature by 100 (say, meters to centimeters) and redo this: the top component snaps to nearly $(0, 1)$ regardless of the correlation structure — units, not signal, chose the axes. That is why you standardize mixed-scale data first.

Cheat sheet

Formulas

• Covariance: $\Sigma = \frac{1}{n-1} X_c^\top X_c$ (centered $X_c$).
• Components: $\Sigma w_i = \lambda_i w_i$; variance along $w_i$ is $\lambda_i$; components orthogonal (spectral theorem).
• Explained variance ratio: $\lambda_i / \sum_j \lambda_j$; keep $k$ by cumulative target, elbow, or CV.
• Via SVD: $X_c = U S V^\top$; directions $= V$, $\lambda_i = s_i^2/(n-1)$, scores $= U S = X_c V$.
• Whitening: divide scores by $\sqrt{\lambda_i}$ → identity covariance (helps some downstream models, amplifies noise dimensions).
• Equivalences: max variance $\iff$ min squared reconstruction error $\iff$ truncated SVD of $X_c$.

Symptom-to-cause table

Gotchas

• PCA is unsupervised: high variance is not the same thing as useful signal, and the top component can be pure nuisance (batch effects, exposure, document length).
• Loadings are directions, not causes — do not over-interpret component "meanings", especially after standardization.
• PCA on one-hot dummies is usually a smell; components destroy the sparsity and the geometry that made the encoding useful.
• With $n < d$ there are at most $n - 1$ nonzero eigenvalues — the rest of the spectrum is exactly zero, not small.
• Correlation-PCA and covariance-PCA give genuinely different components; the choice is a modeling decision, not a numerical detail.