Convolutional neural networks

The idea

A convolution layer slides a small learned kernel across the input, taking a dot product at each position — the same handful of weights, reused everywhere. (Frameworks actually compute cross-correlation, no kernel flip; nobody cares, the weights are learned.) For a kernel $w$ of size $k \times k$ over input patch $x$ centered at $(i, j)$:

$y_{i,j} = b + \sum_{u,v} w_{u,v}\, x_{i+u,\, j+v}$

With multi-channel input, each filter spans the full channel depth ($k \times k \times C_{\text{in}}$ weights) and produces one output channel; $C_{\text{out}}$ filters stack into the output volume.

Two design commitments do all the work:

• Locality: each output looks at a small neighborhood, matching the statistics of images — meaningful structure is local before it is global.
• Weight sharing: one detector is applied at every position, giving translation equivariance for ideal unit-stride convolution away from boundaries. Padding, stride, pooling, and finite edges can break exact equivariance. Parameter count remains independent of image size.

The knobs: stride $s$ (step size; stride 2 halves resolution), padding $p$ (zeros around the border; $p = (k-1)/2$ preserves size at stride 1), and filter count $C_{\text{out}}$. Output size per dimension:

$n_{\text{out}} = \left\lfloor \frac{n + 2p - k}{s} \right\rfloor + 1$

Depth builds a hierarchy: edges → textures → parts → objects. Each layer's receptive field (input region influencing one activation) grows with depth, so late layers reason about large structures using features computed once, locally, by early layers. A standard modern trunk: repeated conv → BN → ReLU blocks, downsample by stride-2 convs a few times, global average pool, one linear classifier.

Worked example

A 1-D convolution you can do in your head. Input $(1, 2, 3, 4, 5)$, kernel $(1, 0, -1)$, stride 1, no padding. Output size: $\lfloor(5 + 0 - 3)/1\rfloor + 1 = 3$.

This kernel is a difference detector — on a constant-slope ramp it reads a constant $-2$ everywhere. Learned 2-D kernels in first layers are exactly this species: oriented edge and gradient detectors.

Parameter arithmetic that wins interviews. CIFAR-scale input $32 \times 32 \times 3$, one conv layer: 16 filters of $5 \times 5$, stride 1, padding 2.

The output shape and parameter count are

A dense layer mapping the same 3,072 inputs to 16,384 outputs would need

a factor of roughly 41,000 more. Same input and output sizes, five orders of magnitude fewer parameters, with position-sharing built in.

Cheat sheet

Formulas

• Output size: $\lfloor (n + 2p - k)/s \rfloor + 1$; same padding at stride 1: $p = (k-1)/2$.
• Parameters: $(k \cdot k \cdot C_{\text{in}} + 1) \cdot C_{\text{out}}$ — independent of input resolution.
• FLOPs scale with output area: $\approx n_{\text{out}}^2 \cdot k^2 \cdot C_{\text{in}} \cdot C_{\text{out}}$ multiply-adds.
• Stacked 3x3 receptive fields: $\ell$ layers see $(2\ell + 1)$ square — 2 layers = 5x5 (18$C^2$ params) vs one 5x5 (25$C^2$); 3 layers = 7x7 (27$C^2$) vs one 7x7 (49$C^2$), plus extra ReLUs.

Components

• Max pool: downsample + local invariance, 0 params; largely replaced by stride-2 convs.
• Global average pool: kills the spatial dims before the classifier → any input size works, no giant flatten layer.
• 1x1 conv: channel mixing / bottlenecks; no spatial context.
• Residual blocks: identity shortcuts make 50-150 layer CNNs trainable (ResNet).
• BN + ReLU after each conv is the default trunk pattern.

Facts and gotchas

• Equivariant to translation, not to rotation or scale — augmentation covers those.
• Channels-out = number of filters; each filter always spans all input channels.
• Stride-2 layer halves H and W → area drops 4x; channel count typically doubles to compensate.
• Edges/textures (early layers) are generic → freeze them when transferring; object-level layers are dataset-specific → retrain those.