Eigenvalues and eigenvectors

The idea

A matrix transforms space: generic vectors get stretched and rotated somewhere new. An eigenvector is a direction the matrix does not rotate — it stays on its own line and is only scaled by the eigenvalue:

$Av = \lambda v, \qquad v \neq 0$

Why this is the single most reused idea in applied linear algebra: along an eigen-direction, the matrix behaves like multiplication by a plain number. If you can find a whole basis of eigenvectors, the matrix is just a list of numbers in the right coordinate system:

$A = Q \Lambda Q^{-1}$

with eigenvectors in the columns of $Q$ and eigenvalues on the diagonal of $\Lambda$ . A coupled $d$ -dimensional linear system becomes $d$ independent one-dimensional ones. Powers collapse to $A^k = Q \Lambda^k Q^{-1}$ , which is why the largest-magnitude eigenvalue controls the fate of every repeated linear process: Markov chains, PageRank, gradient descent dynamics, RNN state propagation. Bigger than one in magnitude → explosion; smaller → decay.

To compute by hand, rearrange $Av = \lambda v$ into $(A - \lambda I)v = 0$ : a nonzero solution exists only when $A - \lambda I$ is singular, giving the characteristic equation

$\det(A - \lambda I) = 0$

For the matrices data scientists actually meet — covariance, correlation, kernel/Gram matrices, graph Laplacians, Hessians — there is a bonus, the spectral theorem: symmetric matrices have real eigenvalues and an orthonormal eigenbasis, $A = Q \Lambda Q^\top$ . Covariance matrices are additionally positive semidefinite ( $w^\top \Sigma w = \mathrm{Var}(w^\top x) \ge 0$ ), so their eigenvalues are nonnegative and readable as variances along perpendicular axes — the fact PCA is built on.

Play with a symmetric matrix directly — the eigenvector rays stay fixed in place as the unit circle stretches into an ellipse around them:

a = 2.0b = 1.0d = 1.0

[[2, 1], [1, 1]] stretches the unit circle along its eigenvector rays by λ₁ = 2.62 and λ₂ = 0.38.

Worked example

Take the symmetric matrix

$A = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}$

Eigenvalues. The characteristic polynomial factors as

\begin{aligned} \det(A - \lambda I) &= (2-\lambda)^2 - 1 \\ &= \lambda^2 - 4\lambda + 3 \\ &= (\lambda - 3)(\lambda - 1). \end{aligned}

Thus $\lambda_1 = 3$ and $\lambda_2 = 1$ . The trace and determinant checks are

\operatorname{tr}(A) = 4 = 3 + 1, \qquad \det(A) = 3 = 3 \times 1.

Eigenvectors. Solve each null-space equation:

\begin{aligned} \lambda_1 = 3:&\quad -v_1 + v_2 = 0 &&\Rightarrow v^{(1)} = (1,1), \\ \lambda_2 = 1:&\quad v_1 + v_2 = 0 &&\Rightarrow v^{(2)} = (1,-1). \end{aligned}

Verification is immediate:

A(1,1) = (3,3) = 3(1,1), \qquad A(1,-1) = (1,-1).

The two eigenvectors are orthogonal, exactly as the spectral theorem promises for a symmetric matrix.

Geometry. $A$ stretches space by $3$ along the diagonal direction $(1,1)$ and leaves the anti-diagonal $(1,-1)$ untouched. If $A$ were a covariance matrix, PCA would report first component $(1,1)/\sqrt{2}$ with variance $3$ and second component $(1,-1)/\sqrt{2}$ with variance $1$ .

Power iteration in miniature. Start from the non-eigenvector $(1, 0)$ and apply $A$ repeatedly:

$(1,0) \to (2,1) \to (5,4) \to (14,13)$

The ratio of coordinates goes $2.0, 1.25, 1.08, \dots \to 1$ : the iterates swing toward the dominant eigen-direction $(1,1)$ , and each step multiplies the components along $(1,1)$ and $(1,-1)$ by $3$ and $1$ respectively, so the dominant one wins at rate $\lambda_2/\lambda_1 = 1/3$ per step.

Cheat sheet

Definitions and hand computation

$Av = \lambda v$ , $v \neq 0$ ; eigenvectors are defined only up to scale (and sign).
Characteristic equation: $\det(A - \lambda I) = 0$ . For $2 \times 2$ : $\lambda^2 - \mathrm{tr}(A)\,\lambda + \det(A) = 0$ .
$\mathrm{tr}(A) = \sum_i \lambda_i$ , $\det(A) = \prod_i \lambda_i$ ; invertible $\iff$ no zero eigenvalue.
Diagonalizable: $A = Q \Lambda Q^{-1}$ ; then $A^k = Q \Lambda^k Q^{-1}$ .
Spectral theorem (symmetric $A$ ): real $\lambda_i$ , orthonormal $Q$ , $A = Q \Lambda Q^\top$ . PSD adds $\lambda_i \ge 0$ ; positive definite, $\lambda_i > 0$ .

Where each fact earns money

Appearance	Eigen-fact doing the work
PCA	covariance eigenvectors = variance axes; eigenvalues = variances
PageRank / power iteration	dominant eigenvector; converges at rate $\lvert \lambda_2/\lambda_1 \rvert$
Markov chains	stationary distribution = left eigenvector for $\lambda = 1$ of $P$ (equivalently the right eigenvector of $P^\top$ ) — the right eigenvector of $P$ for $\lambda = 1$ is just the all-ones vector
Gradient descent / Hessian	positive definite Hessian = local min; $\kappa = \lambda_{\max}/\lambda_{\min}$ sets step-size and convergence speed
Multicollinearity	near-zero covariance eigenvalue = redundant feature combination
Spectral clustering	small eigenvalues of graph Laplacian reveal cluster structure

Gotchas

Non-symmetric real matrices can have complex eigenvalues (rotation matrices: no real invariant direction) or too few eigenvectors to diagonalize at all. The symmetric case is the well-behaved one.
Sample covariance with $n < d$ has rank at most $n - 1$ : expect a pile of exactly-zero eigenvalues, and tiny negative ones from floating point — clip to zero or add a small ridge $\epsilon I$ .
Power iteration stalls when $\lvert \lambda_1 \rvert \approx \lvert \lambda_2 \rvert$ and never leaves the starting subspace if the seed is exactly orthogonal to the dominant eigenvector — restart from a random vector.
Eigenvalues of $A$ are not the singular values of $A$ unless $A$ is symmetric PSD — do not interchange them for rectangular or non-symmetric matrices.