Activation functions · Math Foundry

The idea

The activation function plays two roles at once. Forward, it shapes what a layer can express — it is the only nonlinear ingredient in the network, the thing that stops a deep stack from collapsing into one affine map. Backward, it acts as a per-unit gradient gate: backprop multiplies by $g^{\prime}(z)$ at every layer, so the derivative of your activation gets raised, roughly, to the power of your depth.

That second role decided the history. Classic squashing functions:

$\sigma(z) = \frac{1}{1 + e^{-z}} \in (0,1), \qquad \tanh(z) = \frac{e^z - e^{-z}}{e^z + e^{-z}} \in (-1, 1)$

with derivatives $\sigma^{\prime} = \sigma(1-\sigma) \le 0.25$ and $\tanh^{\prime} = 1 - \tanh^2 \le 1$ . Both saturate: for $|z|$ beyond a few units the derivative is essentially zero, and the unit goes silent in the backward pass. Stack ten such layers and the early ones starve.

ReLU, $\max(0, z)$ , broke the pattern by being piecewise linear: derivative exactly 1 whenever the unit is active — no shrinkage, no saturation on the positive side, trivially cheap. Its one failure mode is the dead unit: pre-activation negative for all inputs means zero output and zero gradient forever. Leaky ReLU ( $\alpha z$ for $z < 0$ , $\alpha \approx 0.01$ ) and GELU ( $z\,\Phi(z)$ , the transformer default) keep a small signal alive on the negative side.

Worked example

How badly does sigmoid throttle the backward pass? Its derivative at a few points:

$z$	$\sigma(z)$	$\sigma^{\prime}(z) = \sigma(1 - \sigma)$
$0$	$0.5$	$0.25$ (the maximum)
$2$	$0.881$	$0.105$
$5$	$0.993$	$0.0066$

Now chain layers together. The activation contributes one factor per layer, so the best possible activation-derivative product through $k$ sigmoid layers (every unit sitting exactly at $z = 0$ ) is $0.25^k$ :

Depth $k$	Sigmoid best case $0.25^k$	ReLU active path
2	$0.0625$	$1$
5	$\approx 0.001$	$1$
10	$\approx 9.5 \times 10^{-7}$	$1$

And that is the optimistic bound — with realistic pre-activations like $z = 2$ the per-layer factor is nearer $0.1$ , giving $10^{-10}$ at depth 10. A gradient that small, multiplied into a sane learning rate, moves first-layer weights by nothing. The same table for ReLU is just 1 at every depth along active paths — which is the honest one-table summary of why deep learning waited for ReLU.

Cheat sheet

Name	Formula	Range	Derivative	Gotcha
Sigmoid	$1/(1 + e^{-z})$	$(0,1)$	$\sigma(1-\sigma) \le 0.25$	saturates both tails; not zero-centered
Tanh	$2\sigma(2z) - 1$	$(-1,1)$	$1 - \tanh^2 \le 1$	still saturates
ReLU	$\max(0, z)$	$[0,\infty)$	$0$ or $1$	dead units
Leaky ReLU	$\max(\alpha z, z)$	$(-\infty,\infty)$	$\alpha$ or $1$	one more hyperparameter ( $\alpha \approx 0.01$ )
GELU	$z\,\Phi(z)$	$\approx (-0.17, \infty)$	smooth, near 0/1	costlier; transformer default
Softmax	$e^{z_i}/\sum_j e^{z_j}$	simplex	—	output probabilities and internal normalized weighting, notably attention

Decision rules

Hidden layers: ReLU by default; GELU in transformers; Leaky ReLU if dead units bite.
Output: linear (regression) / 1 sigmoid (binary) / softmax (one-of-K) / K sigmoids (multi-label).
RNN internals: tanh states, sigmoid gates.
Pair init with activation: He ( $\mathrm{Var} = 2/n_{\text{in}}$ ) for ReLU family, Xavier ( $\mathrm{Var} = 1/n_{\text{in}}$ scale) for tanh/sigmoid.

Symptoms → fixes

Early layers not learning, loss plateaus → saturating activations somewhere; swap to ReLU/GELU, add normalization or residuals.
Many units permanently outputting 0 after LR spike → dead ReLUs; lower LR / warmup, He init, leaky variant.
Multi-label task mysteriously forcing one label per example → softmax where per-label sigmoids belong.