Transfer learning and fine-tuning

The idea

Networks trained on large datasets learn representations whose early and middle layers are strikingly generic — edges and textures in vision, morphology and syntax in language — while only the later layers commit to the pretraining task. Transfer learning is the decision to buy those generic layers instead of re-learning them: start from pretrained weights, and spend your scarce labels only on what is genuinely task-specific.

Two regimes, really a spectrum:

• Feature extraction: freeze the backbone; train a new head (often just a linear layer — a linear probe) on its output features. Trainable parameters drop from millions to thousands, so a few hundred labels suffice, and nothing pretrained can be damaged.
• Fine-tuning: unfreeze some or all layers and keep training at a learning rate 10-100x below scratch scale. More adaptation capacity — needed when your domain differs from the pretraining data — but now you can catastrophically forget: oversized steps overwrite pretrained features faster than a small dataset can re-teach them.

Choosing along two axes — data size and domain similarity:

The standard safe recipe when unfreezing: train the head first (frozen backbone), then unfreeze top-down with discriminative learning rates (each earlier block ~2-10x lower), warmup, early stopping. Modern LLM-scale practice pushes the same logic further with parameter-efficient fine-tuning: LoRA freezes $W$ and learns a low-rank $\Delta W = BA$ (rank 4-64, ~0.1-1% of parameters), mergeable into $W$ after training — adaptation without touching, or storing copies of, the base model.

Worked example

The "500 labeled images, 5 classes" scenario, in numbers, with an ImageNet-pretrained ResNet-50 (25.6M parameters, 2048-d final features):

The head-only row is a 5-class logistic regression on 2048-d pretrained features. It is a sensible low-variance starting point, not something that 500 labels automatically guarantee. Multiply the data with augmentation (flips, crops, color jitter), pick the stopping epoch by validation loss, and use 5-fold cross-validation for reporting, since one 100-image validation fold has a standard error of several percent on its own. Only if the probe plateaus below target: unfreeze the final stage (~15M of the 25.6M parameters sit there) at one-tenth the head LR for a few epochs — the classic two-phase recipe.

Cheat sheet

• Order of attack with scarce labels: linear probe → partial unfreeze (top-down) → full fine-tune → (LLMs) LoRA instead of full.
• LRs: fine-tuning often starts 10-100x below scratch; layer-wise decay and warmup are tunable defaults, not universal requirements.
• BERT-class defaults: AdamW, LR 2e-5 to 5e-5, 2-4 epochs, batch 16-32, linear warmup + decay; multiple seeds on small data.
• Vision defaults: replace head; augment aggressively; freeze BN running stats (or whole BN layers) when batches are small.
• 2x2 rule: small+similar → head only; small+different → tune top layers; large+similar → full FT low LR; large+different → full FT vs scratch baseline.
• Catastrophic forgetting fixes: lower LR, gradual unfreezing, early stopping, layer-wise LR decay, LoRA/adapters (base frozen by construction).
• LoRA numbers: rank $r$ 4-64; $\Delta W = BA$ adds $r(d_{\text{in}} + d_{\text{out}})$ params per matrix (~0.1-1% total); merge $BA$ into $W$ post-training → zero inference overhead.
• Evidence checks: strong linear probe → features suffice; scratch baseline ties fine-tuned → transfer added nothing; fine-tuned worse than probe → you forgot faster than you learned (LR too high, too many epochs).
• Gotcha: never report accuracy from the split you used for early stopping / model picking — nested or repeated CV when data is this small.
• Gotcha: re-embedding rule applies here too — swap the backbone and every cached downstream feature is invalid.