Overfitting and generalization

The idea

Training error is a rigged benchmark: the model was optimized on exactly those points, so it will always look better there than on the population it came from. Generalization is the only thing that pays — expected performance on new data drawn from the same distribution. Overfitting is what happens when optimizing the rigged benchmark starts to hurt the real one: the model uses its capacity to encode the noise and coincidences of one particular sample.

The operational quantity is the generalization gap:

$\text{gap} = \mathbb{E}[L_{\text{test}}] - L_{\text{train}}$

A small gap with high error means underfitting (bias); a large gap means overfitting (variance). Since expected test loss is unobservable, you estimate it with held-out data, and the protocol matters:

• Train set — the model fits here. Error here is optimistic, always.
• Validation set — you tune here: hyperparameters, feature choices, early stopping. Because you make many decisions against it, its error drifts optimistic too.
• Test set — touched once, at the end, to report the final number. Every additional peek converts a little of it into validation data.

The most useful diagnostic plot is the learning curve: train and validation error as a function of training-set size. Converged-and-high means bias (more data will not help); a persistent gap that shrinks with data means variance (more data will help). This one plot answers the most expensive question in applied ML — should we go collect more data? — before you spend the money.

The failure mode that mimics great generalization is leakage: the model sees information at training time that will not exist at prediction time. Leakage produces beautiful offline metrics and a production disaster, which is why "too good to be true" validation scores should trigger a pipeline audit, not a celebration.

Worked example

A gradient-boosted classifier is trained on subsets of increasing size; error rates (1 - accuracy) on the training subset and a fixed validation set:

Reading it like an interviewer wants:

• At $n = 100$ the model memorizes perfectly (train error 0.00) and generalizes terribly — pure variance.
• As data grows, training error rises (harder to memorize 20,000 points than 100) while validation error falls, and the two converge near 0.10.
• The curves meeting around 0.10-0.11 says variance is nearly exhausted; if 0.11 is not good enough, more rows will not fix it — you now need better features or a different model class, i.e. you are fighting bias.

Contrast: if the table had shown train 0.02 / val 0.19 at 20,000 rows with the gap still shrinking, the correct move would be the opposite — collect more data or regularize harder.

Cheat sheet

Gap: $\text{gap} = L_{\text{val}} - L_{\text{train}}$; overfitting = low train error + large gap; underfitting = high train error + small gap.

• Anti-overfitting toolkit: more data, L1/L2 penalties, capacity limits (depth, features), early stopping, bagging, dropout (neural nets), data augmentation.
• Fit ALL data-dependent preprocessing (scaling, imputation, encoding, feature selection, resampling) on the training fold only — use a pipeline so cross-validation cannot cheat.
• Validation error is optimistically biased after heavy tuning — that is why the untouched test set exists. Report test once.
• Leakage red flags: features recorded after the label event, duplicate rows across splits, group structure ignored (same user in train and test), temporal data split randomly instead of chronologically.
• Overfitting is not binary — every fitted model overfits a little; the question is whether the gap is large relative to the error you care about.