Fundamentals

What sigmoid, tanh, ReLU, and GELU each do to forward signals and backward gradients — and why the default answer is the ReLU family

The chain rule organized so one backward sweep computes the gradient of the loss with respect to every parameter, reusing shared work at each step

The full treatment of batch norm and layer norm: what normalization layers compute, why batch norm behaves differently at inference, and why transformers standardized on layer norm

How dropout, weight decay, early stopping, and augmentation keep overparameterized networks honest — plus the batch-norm and layer-norm facts to have on hand

How a network turns inputs into predictions — alternating affine maps and nonlinearities, with every intermediate cached for the backward pass

The ladder from vanilla SGD through momentum to Adam and AdamW — what each rung adds, with the update rules and defaults worth memorizing