Deep Learning

What sigmoid, tanh, ReLU, and GELU each do to forward signals and backward gradients — and why the default answer is the ReLU family

The chain rule organized so one backward sweep computes the gradient of the loss with respect to every parameter, reusing shared work at each step

The full treatment of batch norm and layer norm: what normalization layers compute, why batch norm behaves differently at inference, and why transformers standardized on layer norm

How dropout, weight decay, early stopping, and augmentation keep overparameterized networks honest — plus the batch-norm and layer-norm facts to have on hand

How a network turns inputs into predictions — alternating affine maps and nonlinearities, with every intermediate cached for the backward pass

The ladder from vanilla SGD through momentum to Adam and AdamW — what each rung adds, with the update rules and defaults worth memorizing

Scaled dot-product attention as a differentiable soft lookup — queries score keys, softmax turns scores into weights, the output is a weighted mix of values

Locality and weight sharing turn images into a parameter-efficient feature hierarchy — plus the output-size and parameter-count formulas interviewers make you compute

Vanilla recurrence and backpropagation through time, why gradients vanish or explode across long sequences, and how gated cells (LSTM, GRU) restore long-range learning

Multi-head attention plus position-wise FFNs, residuals, and positional encodings — and the parallelism argument that retired recurrent networks

Learned dense vectors that place discrete things in a geometry where distance means similarity — the representation trick under NLP, recsys, and semantic search

Reusing pretrained representations — when to freeze the backbone, when to fine-tune, and how to make 500 labeled examples go a very long way