RNNs & LSTMs

The idea

An RNN reads a sequence one element at a time and folds it into a fixed-size hidden state — a running, lossy summary of everything seen so far:

$h_t = \tanh(W_h h_{t-1} + W_x x_t + b)$

The same $W_h$, $W_x$, $b$ are reused at every step: weight sharing across time, exactly as a CNN shares weights across space. That makes the model length-agnostic and keeps parameters at $O(d^2)$ regardless of sequence length — but it also means one matrix, $W_h$, sits on every path from the past to the present.

Backpropagation through time (BPTT). To train, unroll the recurrence into a deep feedforward net with one layer per step (all layers sharing weights), backpropagate, and sum each weight gradient over every step where the weight was used. The chain rule from step $t$ back to step $k$ multiplies one recurrent Jacobian per step:

$\frac{\partial h_t}{\partial h_k} = \prod_{j=k+1}^{t} \frac{\partial h_j}{\partial h_{j-1}}, \qquad \frac{\partial h_j}{\partial h_{j-1}} = \mathrm{diag}\!\left(1 - h_j^2\right) W_h$

Why gradients vanish or explode. A product of $t - k$ near-identical matrices scales like powers of the largest singular value $\sigma_{\max}(W_h)$: at $0.9$ per step, 50 steps leave a factor $0.9^{50} \approx 0.005$; at $1.1$, they produce $1.1^{50} \approx 117$. So gradients from distant steps either drown (vanish — the model simply never learns long-range dependencies, and trains fine otherwise, which is what makes it insidious) or blow up (explode — loss spikes, NaNs). Tanh saturation compounds the vanishing side: the $\mathrm{diag}(1 - h_j^2)$ factor is at most 1 and near 0 for saturated units, dragging the effective per-step factor below $\sigma_{\max}$.

Fix for explosion: clip. Rescale the gradient when its global norm exceeds a threshold $\theta$ (typically 1–5): $g \leftarrow \theta \, g / \|g\|$ if $\|g\| > \theta$. Direction preserved, step size capped. Cheap, standard — and useless against vanishing, since you cannot rescale a signal that never arrived.

Fix for vanishing: the LSTM. Add a second state, the cell $c_t$, with an additive update controlled by three sigmoid gates (each a function of $h_{t-1}$ and $x_t$):

Forget gate $f_t$: what fraction of old memory to keep. Input gate $i_t$: how much of the new candidate $\tilde{c}_t$ to write. Output gate $o_t$: how much of the memory to expose as $h_t$. The gradient payoff is the direct path through the cell: $\partial c_t / \partial c_{t-1} = f_t$ — no $W_h$, no tanh derivative. When the network holds $f_t$ near 1, gradients travel back essentially unchanged: a gated highway through time. (Residual connections are the same additive trick with the gate hard-wired to 1.) Practical corollary: initialize the forget-gate bias to 1–2 so the highway starts open.

GRU: the 2-gate budget option. Merge cell and hidden state, keep an update gate $z_t$ and a reset gate $r_t$:

Three gated blocks instead of four, so about 3/4 the parameters, often matching LSTM quality. (Convention gotcha: some sources swap the roles of $z_t$ and $1 - z_t$ — check before comparing code to a paper.)

What gating never fixed. Training is still sequential — $h_t$ waits for $h_{t-1}$, so no parallelism across positions, which is what GPUs want — and information $n$ steps back must still survive $n$ updates through a fixed-size state, so effective context stays far below nominal context. Transformers solved both: all positions train in parallel, and attention gives an $O(1)$-hop path between any two tokens. Recurrence remains sensible where its strengths bind: streaming/online inference (constant-size state, $O(1)$ per step vs a growing KV cache), small data, constrained compute, and some time series. Modern state-space models (Mamba and kin) revisit exactly this trade with parallelizable training.

Worked example

One LSTM step with scalar states, so every number can be checked by hand. Previous cell $c_{t-1} = 1.0$, and suppose the gate pre-activations (the values of $W [h_{t-1}, x_t] + b$) come out as:

(Check: $\sigma(2) = 1/(1 + e^{-2}) = 1/1.135 = 0.881$; $\sigma(0) = 0.5$; $\tanh(1) = 0.762$; $\sigma(1) = 1/1.368 = 0.731$.)

New cell state — keep 88.1% of the old memory, write half of the candidate:

$c_t = f_t \cdot c_{t-1} + i_t \cdot \tilde{c}_t = 0.881 \times 1.0 + 0.500 \times 0.762 = 0.881 + 0.381 = 1.262$

New hidden state — squash the cell, then let the output gate meter it out:

$h_t = o_t \cdot \tanh(c_t) = 0.731 \times \tanh(1.262) = 0.731 \times 0.852 = 0.623$

Note what did not happen to $c$: it was never multiplied by a recurrent weight matrix or pushed through a saturating nonlinearity — only scaled by $f_t = 0.881$ and nudged additively. The backward derivative along that path is exactly $0.881$.

Gradient survival 20 steps back, direct path only, per-step factor to the 20th power:

The vanilla signal is gone; the open-gate LSTM keeps 82% of it. But note $0.881^{20} \approx 0.08$: LSTMs mitigate vanishing, they do not repeal it — any forget gate below 1 still decays geometrically, just with a base the network controls.

Cheat sheet

Vanilla RNN: $h_t = \tanh(W_h h_{t-1} + W_x x_t + b)$ — same weights every step; hidden state = fixed-size running summary.

BPTT: unroll, backprop, sum gradients over steps; $\partial h_t / \partial h_k$ is a product of Jacobians $\mathrm{diag}(1 - h_j^2)\, W_h$, so it scales like $\sigma_{\max}(W_h)^{t-k}$ (tanh saturation pushes the effective base lower).

LSTM: $f_t, i_t, o_t = \sigma(\cdot)$, $\tilde{c}_t = \tanh(\cdot)$; $c_t = f_t \odot c_{t-1} + i_t \odot \tilde{c}_t$; $h_t = o_t \odot \tanh(c_t)$. Gradient highway: $\partial c_t / \partial c_{t-1} = f_t$.

• Clipping fixes explosion only — a vanished gradient cannot be rescaled back into existence.
• LSTMs mitigate, not solve: per-step survival is $f_t$; $0.99^{20} \approx 0.82$ but $0.9^{50} \approx 0.005$.
• $h_t$ is the exposed working memory; $c_t$ is the protected long-term memory — do not conflate them.
• vs transformers: recurrence trains sequentially ($O(n)$ steps, no position parallelism) with $O(n)$-hop gradient paths; attention trains in parallel with $O(1)$-hop paths but pays $O(n^2)$ compute and a growing KV cache. RNN inference state stays $O(1)$.
• Still reach for RNN/LSTM: streaming/online inference, small datasets, edge/latency budgets, some time series. State-space models (e.g. Mamba) are the modern recurrent revival.
• GRU convention gotcha: whether $z_t$ multiplies the old state or the candidate differs across papers/frameworks.