Bias-variance tradeoff

The idea

Under squared-error loss, a model's expected prediction error at a point can be split into three pieces that add together:

$\mathbb{E}[(y - \hat{f}(x))^2] = \text{Bias}[\hat{f}(x)]^2 + \text{Var}[\hat{f}(x)] + \sigma^2$

• Bias is how far the model's average prediction, taken over many hypothetical training sets, sits from the true function: $\text{Bias}[\hat{f}(x)] = \mathbb{E}[\hat{f}(x)] - f(x)$. High bias means the model class is too restrictive to represent the pattern — a straight line trying to fit a curve, or a shallow tree trying to separate classes that need several splits.
• Variance is how much the prediction moves when you retrain on a different sample from the same distribution: $\text{Var}[\hat{f}(x)] = \mathbb{E}[(\hat{f}(x) - \mathbb{E}[\hat{f}(x)])^2]$. High variance means the model is chasing the noise in one particular training set rather than the underlying signal.
• Irreducible noise, $\sigma^2$, is the floor imposed by randomness in the data-generating process itself. No amount of modeling removes it.

You never observe bias and variance directly — you only ever see one fitted model and its error on one test set. What you can observe is the gap between training error and test error, and that gap is your best working proxy: a model with training error close to test error (both mediocre) is underfitting and needs more capacity; a model with training error far below test error is overfitting and needs less capacity, more data, or more regularization. Increasing model flexibility — deeper trees, more features, weaker regularization, a higher-degree polynomial — typically pushes bias down and variance up. Finding the sweet spot on that curve is what "model selection" means in practice.

Worked example

Fit polynomials of increasing degree to noisy data generated from a smooth underlying curve with irreducible noise variance $\sigma^2 = 0.01$. Measured mean squared error on a held-out test set looks like this:

Reading the table the way an interviewer wants:

• From degree 1 to degree 3, both train and test error drop sharply — the model was too simple, so adding capacity reduced bias with only a small variance cost.
• From degree 3 to degree 9, train error keeps falling toward zero, but test error gets worse. The extra capacity is no longer tracking signal; it is memorizing the particular noise draws in this training sample. The growing train/test gap is the variance term taking over.
• Degree 3 is close to the true complexity of the generating process here, so its test error (0.11) is not far above the noise floor ($\sigma^2 = 0.01$) plus the small remaining bias — there just is not much room left to improve without a fundamentally different feature set.

The shape to remember: test error traces a U as capacity increases — bias dominates the left arm, variance dominates the right arm, and the minimum sits wherever their sum is smallest.

Cheat sheet

Formula: $\mathbb{E}[(y-\hat{f}(x))^2] = \text{Bias}[\hat{f}(x)]^2 + \text{Var}[\hat{f}(x)] + \sigma^2$

• k-NN: small $k$ → low bias, high variance (jagged decision boundary chases every neighbor). Large $k$ → high bias, low variance (smoother, more averaged boundary).
• Decision trees: deep, unconstrained trees → low bias, high variance. Shallow trees → high bias, low variance.
• Regularization strength $\lambda$: increasing $\lambda$ increases bias and decreases variance; it is a direct dial on this tradeoff.
• Bagging (e.g. random forests) mainly reduces variance, by averaging many decorrelated high-variance learners, without increasing bias much.
• Boosting mainly reduces bias, by sequentially fitting residuals with weak learners, though it can raise variance if left unchecked.
• More training data reduces variance but does nothing for bias — a model that is too simple stays too simple no matter how much data you feed it.
• Underfitting and high bias are the same idea from two angles; overfitting and high variance are the same idea from two angles.