Backpropagation · Math Foundry

The idea

Training needs $\partial L/\partial \theta$ for every parameter $\theta$ , and a modern network has millions of them. Backprop is the observation that all those derivatives share structure. The chain rule for a composition $L = f_3(f_2(f_1(x)))$ multiplies local derivatives along the path — and if you evaluate that product from the loss backwards, every parameter's gradient reuses the partial products already computed for the layers above it. One backward sweep—same asymptotic order as the forward pass and often roughly twice its FLOPs in dense networks—produces all parameter gradients.

The bookkeeping runs on the error signal $\delta^{(\ell)} = \partial L/\partial z^{(\ell)}$ (loss gradient with respect to layer $\ell$ 's pre-activation). For a dense layer $z^{(\ell)} = W^{(\ell)} a^{(\ell-1)} + b^{(\ell)}$ , three local rules do everything:

\begin{aligned} \frac{\partial L}{\partial W^{(\ell)}} &= \delta^{(\ell)} \left(a^{(\ell-1)}\right)^\top, \\ \frac{\partial L}{\partial b^{(\ell)}} &= \delta^{(\ell)}, \\ \frac{\partial L}{\partial a^{(\ell-1)}} &= W^{(\ell)\top} \delta^{(\ell)}. \end{aligned}

The third rule is the recursion: it hands the error to the previous layer, which multiplies elementwise by its activation derivative, $\delta^{(\ell-1)} = W^{(\ell)\top}\delta^{(\ell)} \odot g^{\prime}\!\left(z^{(\ell-1)}\right)$ , and repeats. Note the first rule needs $a^{(\ell-1)}$ — this is exactly why the forward pass caches every layer input.

Two consequences fall straight out of the product structure:

Vanishing gradients: if each layer's factor has magnitude below 1 (saturated sigmoids contribute at most $0.25$ ), the product decays exponentially with depth and early layers stop learning.
Exploding gradients: factors above 1 compound the same way upward; the symptom is loss spikes and NaNs, the blunt fix is gradient clipping.

Worked example

The interview classic — one hidden ReLU unit, sigmoid output, and BCE loss — uses:

\begin{aligned} x &= 2, & y &= 1, \\ w_1 &= 0.5, & b_1 &= -0.5, \\ w_2 &= 2, & b_2 &= -0.5. \end{aligned}

Forward (cache everything):

\begin{aligned} z_1 &= 0.5 \cdot 2 - 0.5 = 0.5, \\ a_1 &= 0.5, \\ z_2 &= 2 \cdot 0.5 - 0.5 = 0.5, \\ \hat{y} &= \sigma(0.5) \approx 0.6225. \end{aligned}

The loss is:

L = -\ln \hat{y} \approx 0.474.

Backward, one row per chain-rule hop:

Quantity	Rule	Value
$\delta_2 = \partial L/\partial z_2$	$\hat{y} - y$	$-0.3775$
$\partial L/\partial w_2$	$\delta_2 \cdot a_1$	$-0.1888$
$\partial L/\partial b_2$	$\delta_2$	$-0.3775$
$\partial L/\partial a_1$	$\delta_2 \cdot w_2$	$-0.7550$
$\delta_1 = \partial L/\partial z_1$	$\partial L/\partial a_1 \cdot \mathbf{1}[z_1 > 0]$	$-0.7550$
$\partial L/\partial w_1$	$\delta_1 \cdot x$	$-1.5100$
$\partial L/\partial b_1$	$\delta_1$	$-0.7550$

Sanity checks worth saying out loud in an interview: every gradient is negative, so a gradient-descent step increases every parameter, which pushes $z_2$ up and $\hat{y}$ toward the label $y = 1$ — the signs make sense. And after one SGD step with $\eta = 0.1$ (e.g. $w_1 \leftarrow 0.651$ , $w_2 \leftarrow 2.019$ ), rerunning the forward pass gives $\hat{y} \approx 0.787$ and $L \approx 0.239$ : the loss halved.

Had $z_1$ been negative, $\mathbf{1}[z_1 > 0] = 0$ would have killed $\delta_1$ and both first-layer gradients — the "dead ReLU" failure mode in miniature: no activation, no gradient, no learning on that path.

Cheat sheet

Core recursion: starting from $\delta^{(L)}$ , repeat for each layer:

\begin{aligned} \frac{\partial L}{\partial W} &= \delta a_{\text{in}}^\top, \\ \frac{\partial L}{\partial b} &= \delta, \\ \delta_{\text{previous}} &= W^\top\delta \odot g'(z). \end{aligned}

Op	Local backward rule
Dense $z = Wa + b$	$dW = \delta a^\top$ , $db = \delta$ , $da = W^\top \delta$
ReLU	pass $\delta$ where $z > 0$ , zero elsewhere
Sigmoid (generic)	$\delta \cdot \sigma(z)(1 - \sigma(z))$
Sigmoid + BCE (fused)	$\hat{y} - y$ directly at the logit
Softmax + CE (fused)	$p - y_{\text{onehot}}$ directly at the logits
Residual $y = x + f(x)$	gradients split and add at the junction: identity branch passes $\delta$ unchanged, residual branch contributes $f^{\prime}(x)^\top \delta$ — the identity term is the gradient highway
Branch reused twice	gradients from both consumers sum

Cost: backward ≈ 2x forward FLOPs; total training step ≈ 3x forward.
Memory: all layer inputs cached; gradient checkpointing recomputes activations to cut memory ~ $O(\sqrt{L})$ at ~1 extra forward pass.
Vanishing: symptoms — early-layer gradients orders of magnitude below late-layer ones, loss plateaus. Fixes — ReLU/GELU, He/Xavier init, batch/layer norm, residual connections.
Exploding: symptoms — loss spikes/NaN. Fixes — gradient clipping (clip global norm to ~1.0), lower LR, better init.
Gradient check: central difference $(L(\theta + \varepsilon) - L(\theta - \varepsilon))/(2\varepsilon)$ , float64, eval mode, compare relative error (pass ≈ $10^{-7}$ ).
$\hat{y} - y$ at the output logit is the single most reusable fact in gradient interviews.