L1 and L2 regularization

The idea

An unregularized linear model will use any freedom it has — including noise directions — to shave training error. Regularization charges the model rent for coefficient size, so it only keeps weights the data genuinely pay for:

Both shrink coefficients toward zero and trade a little bias for a lot of variance reduction. The interview-famous difference: lasso sets coefficients exactly to zero; ridge only shrinks them. Two ways to see why:

Geometry. Penalized form is equivalent to minimizing RSS subject to a budget $\|\beta\|_1 \le t$ (a diamond) or $\|\beta\|_2^2 \le t$ (a disk). The solution sits where the elliptical RSS contours first touch the budget region. A diamond has corners on the axes, and expanding ellipses tend to strike a corner first — and at a corner, some coordinates are exactly zero. A disk has no corners, so the tangent point is generically off-axis: every coefficient survives, smaller.

Subgradients. At $\beta_j = 0$, the L1 term contributes any value in $[-\lambda, \lambda]$ to the gradient. If the data's pull on that coefficient satisfies $|\partial \text{RSS}/\partial \beta_j| \le \lambda$, zero is optimal and stays optimal — zero is sticky. The ridge penalty's force is $\lambda\beta_j$, which vanishes as $\beta_j \to 0$: it can push a coefficient arbitrarily close to zero but never hold it there.

Bayesian reading: ridge is MAP estimation under a Gaussian prior on $\beta$; lasso under a Laplace prior, whose sharp peak at zero encodes the belief "most coefficients are exactly irrelevant."

Practicalities that interviewers probe: standardize features first (the penalty is unit-blind, coefficients are not), do not penalize the intercept, and remember lasso's instability on correlated features — it picks one member of a correlated group nearly arbitrarily, while ridge splits weight across the group. Elastic net, $\lambda_1\|\beta\|_1 + \tfrac{\lambda_2}{2}\|\beta\|_2^2$, buys sparsity and grouping together.

Worked example

With an orthonormal design ($X^{\top}X = I$) the solutions are closed-form, so you can watch both penalties act on the same OLS fit. Take $\hat{\beta}^{\text{OLS}} = (3.0,\; 0.4,\; -1.2)$ and $\lambda = 0.5$:

Ridge divides everything by $1 + \lambda = 1.5$:

$\hat{\beta}^{\text{ridge}} = (2.0,\; 0.27,\; -0.8)$

Lasso soft-thresholds — move each coefficient $0.5$ toward zero, clip at zero:

$\hat{\beta}^{\text{lasso}} = (2.5,\; 0,\; -0.7)$

The story in one row: $\beta_2 = 0.4$ is weaker than the threshold $\lambda = 0.5$, so lasso deletes it outright, while ridge keeps it at $0.27$ — nonzero forever, no matter how large $\lambda$ grows, it only approaches zero asymptotically. Also note ridge punished the largest coefficient hardest in absolute terms (3.0 lost 1.0; 0.4 lost 0.13): proportional shrinkage, versus lasso's flat "everyone pays $0.5$".

Cheat sheet

Objectives: lasso $\tfrac{1}{2}\|y-X\beta\|_2^2 + \lambda\|\beta\|_1$; ridge $\tfrac{1}{2}\|y-X\beta\|_2^2 + \tfrac{\lambda}{2}\|\beta\|_2^2$; elastic net adds both.

Orthonormal closed forms: ridge $\hat{\beta}_j / (1+\lambda)$; lasso $\text{sign}(\hat{\beta}_j)\max(|\hat{\beta}_j| - \lambda, 0)$.

• $\lambda \uparrow$ = bias $\uparrow$, variance $\downarrow$; $\lambda = 0$ is OLS; $\lambda \to \infty$ kills all penalized coefficients.
• Standardize features first; never penalize the intercept.
• Ridge fixes the singular / near-singular $X^{\top}X$ problem: $(X^{\top}X + \lambda I)$ is always invertible — this is why ridge is the standard answer to multicollinearity.
• Choose $\lambda$ by CV on a log grid; the one-standard-error rule trades a whisker of CV score for a sparser, more robust model.
• Elastic net default when $p \gg n$ or features come in correlated blocks.
• Gotcha: lasso can select at most $\min(n, p)$ features — relevant when $p \gg n$.