Dropout and regularization

The idea

Modern networks have enough parameters to memorize their training sets outright, so generalization comes from constraining how they fit — regularization. The deep-learning toolkit has one method with a genuinely novel mechanism (dropout), plus classical tools carried over.

Dropout. During training, each unit is independently zeroed with probability $p$ ; survivors are scaled by $1/(1-p)$ (this is inverted dropout, what every framework implements). Formally, with mask entries $m_i \sim \mathrm{Bernoulli}(1-p)$ :

$\tilde{a}_i = \frac{m_i}{1-p}\, a_i, \qquad \mathbb{E}[\tilde{a}_i] = \frac{(1-p)}{1-p} a_i = a_i$

The expectation is preserved, so at test time you simply switch dropout off and run a normal forward pass. Why it works, two complementary stories: (1) no co-adaptation — a unit cannot depend on a specific other unit existing, so features become individually useful and redundant; (2) implicit ensemble — every mask is a different subnetwork with shared weights, and the test-time network approximates averaging exponentially many of them.

The rest of the toolkit: weight decay (penalize $\|\theta\|^2$ ; use AdamW so it is not mangled by adaptive scaling), early stopping (halt when validation loss turns; free and always on), data augmentation (label-preserving transforms — the highest-leverage option for images and audio), and label smoothing (soften the one-hot target to $1-\alpha+\alpha/K$ for the correct class — $0.91$ at $\alpha=0.1$ , $K=10$ — and $\alpha/K$ for each wrong class; discourages overconfident logits).

Batch/layer normalization sit adjacent: they exist to stabilize optimization, not to regularize, but BN's minibatch noise has a mild regularizing side effect, and their mechanics are tested in the same interviews — key facts in the cheat sheet.

Worked example

A hidden layer outputs $a = (2, -1, 3, 0.5)$ with dropout $p = 0.5$ . One training step draws mask $m = (1, 0, 1, 0)$ :

$\tilde{a} = \frac{(1, 0, 1, 0) \odot (2, -1, 3, 0.5)}{1 - 0.5} = (4,\; 0,\; 6,\; 0)$

Surviving units are doubled; dropped units contribute nothing (and get no gradient this step — the mask gates the backward pass identically). The next minibatch draws a fresh mask, say $m = (0, 1, 1, 1)$ , giving $\tilde{a} = (0, -2, 6, 1)$ : a different subnetwork now takes the gradient step, though it shares every weight with the first. Training is effectively a lottery over $2^4 = 16$ subnetworks of this layer — that is the ensemble picture in miniature.

Check the expectation for the first unit across the mask distribution: it survives with probability $0.5$ at value $4$ , else $0$ — expected value $0.5 \cdot 4 = 2$ , exactly the original activation. Downstream weights therefore see the same average input during training as they will at test time, when the layer just emits $(2, -1, 3, 0.5)$ untouched. Without the $1/(1-p)$ scaling, training-time activations would average half the test-time ones, and the network would arrive at inference systematically miscalibrated.

Cheat sheet

Dropout

Train: zero with prob $p$ , scale survivors by $1/(1-p)$ . Test: no-op.
Typical $p$ : 0.5 classic dense heads; 0.1-0.3 modern nets; 0.1 in transformers (attention, FFN, embeddings).
Skip it on: conv trunks (use BN + augmentation), right before BN, output layers.
Nonstandard uses: MC dropout at test time = cheap uncertainty estimates.

Norm layers (the facts interviews want)

	Batch norm	Layer norm
Normalizes over	batch, per channel	features, per example
Formula core	$(x - \mu_B)/\sqrt{\sigma_B^2 + \epsilon}$ , then $\gamma, \beta$	same, with per-example $\mu, \sigma$
Train vs eval	different (batch stats vs running stats)	identical
Small/variable batches	breaks below ~8	unaffected
Home turf	convnets	transformers, RNNs

BN classic bug: eval in train mode (or vice versa) → metrics nonsense; check model.eval() first.
BN regularizes mildly via batch-stat noise; do not count it as your regularizer.

Other regularizers

Weight decay: AdamW $\lambda \approx 0.01$ - $0.1$ (transformers), $10^{-4}$ scale for SGD vision recipes.
Early stopping: monitor val loss, patience a few evals; cheapest win available.
Augmentation: images — flips/crops/color/mixup; text — harder, use paraphrase/backtranslation sparingly.
Label smoothing: $\alpha = 0.1$ standard for classification.

Symptom → move: big train/val gap → augment + early stop + decay + dropout, in that order; train accuracy also low → you have a bias problem, regularizing makes it worse.