Forward pass

The idea

A feedforward network is just function composition: the input is pushed through a stack of layers, each of which does something very simple, and the prediction pops out the far end. For a dense (fully connected) layer $\ell$:

$z^{(\ell)} = W^{(\ell)} a^{(\ell-1)} + b^{(\ell)}, \qquad a^{(\ell)} = g\left(z^{(\ell)}\right)$

• $a^{(\ell-1)}$ is the previous layer's output (with $a^{(0)} = x$, the input),
• $W^{(\ell)}$ and $b^{(\ell)}$ are the layer's learned weights and biases,
• $g$ is an elementwise nonlinearity such as ReLU,
• $z$ is called the pre-activation and $a$ the activation.

Two things make this composition interesting. First, the nonlinearity: without $g$, the whole stack collapses into one affine map, so a 1000-layer linear network is exactly a linear model. Second, the caching: every intermediate $a^{(\ell)}$ (and often $z^{(\ell)}$) is kept in memory, because backpropagation reuses them — the forward pass is not just computing a prediction, it is laying down the breadcrumbs the backward pass will follow.

The final layer is a task-specific head. Its raw outputs are logits — unbounded real scores — and the last step maps them to the right output space: identity for regression, sigmoid for a binary probability, softmax $p_i = e^{z_i}/\sum_j e^{z_j}$ for a distribution over classes.

In practice everything is batched: $N$ inputs are stacked into a matrix $X$ of shape $N \times d$, and each layer transforms the whole batch in one matrix multiply. That is the entire reason GPUs make deep learning feasible.

Worked example

A 2-input, 2-hidden-unit (ReLU), 1-output (sigmoid) network, with

Push $x = (1, 2)$ through it:

The model says "class 1 with probability 7.6%". Note what happened at the ReLU: the first hidden unit's pre-activation was negative, so it output exactly 0 and contributed nothing downstream — and, as backprop will make precise, no gradient will flow through it for this example either. The cached values $x$, $z^{(1)}$, $a^{(1)}$, $z^{(2)}$ are exactly what the backward pass consumes.

Parameter count, using layer shape $d_{\text{out}} \times d_{\text{in}} + d_{\text{out}}$: layer 1 has $2 \times 2 + 2 = 6$, layer 2 has $1 \times 2 + 1 = 3$, so this network has 9 trainable parameters. The same bookkeeping at scale — count weights plus biases per layer, sum over layers — is a standard interview warm-up, and it is worth noticing that a batch of 64 such inputs would change none of it: parameters are shared across the batch, only the activation tensors grow.

Cheat sheet

• Layer: $z = Wx + b$, $a = g(z)$. $W$ has shape $d_{\text{out}} \times d_{\text{in}}$ — parameters per dense layer: $d_{\text{out}} \cdot d_{\text{in}} + d_{\text{out}}$.
• Batch form: $X$ is $N \times d_{\text{in}}$, layer computes $XW^\top + b$ → $N \times d_{\text{out}}$. Bias broadcasts over the batch dimension.
• No nonlinearity → no depth: any stack of affine layers equals one affine layer.
• Softmax: $p_i = e^{z_i} / \sum_j e^{z_j}$; subtract $\max_j z_j$ first for numerical stability (frameworks do this for you).

• Memory: training caches every layer input for backprop → activation memory scales with batch size × network width × depth; inference does not.
• Train vs eval mode: dropout on / batch-norm batch-stats during training; dropout off / running stats at eval. Forgetting to switch is a classic silent bug.
• Gotcha: compute losses from logits (fused log-sum-exp), never log(softmax(z)) by hand.
• Gotcha: a ReLU unit that outputs 0 passes zero gradient — visible already in the forward pass as inactive units.