Convolutions with MNIST

A convolution is the operation at the heart of every CNN, and it is simpler than it sounds: slide a small grid of numbers over an image, multiply, add up. Work through the four steps below on a real MNIST digit and watch a feature map appear.

What just happened

The kernel is a little window of weights. It is placed over the top-left corner of the image, each weight is multiplied by the pixel underneath it, and those products are added into a single number. The window then shifts one pixel to the right and the whole thing repeats — that shift is called the stride, here 1. Hover over any cell of the feature map above and the exact sum that produced it is spelled out.

Because a 3 × 3 window cannot be centred on the outermost pixels, the output comes out slightly smaller than the input: 28 × 28 becomes 26 × 26, and with a 5 × 5 kernel 24 × 24. Real networks often add a border of zeros — padding — to keep the size unchanged.

Look at what the different presets do. The vertical kernel has positive weights on its left and negative ones on its right, so it produces a large number wherever the image is bright on the left and dark on the right — a vertical edge — and roughly zero across flat areas, where the positives and negatives cancel. The horizontal kernel is the same idea rotated. Blur just averages the neighbourhood. Every one of these is doing the same arithmetic; only the numbers differ.

Why this matters for CNNs

Everything above was hand-picked, and that is the one misleading part of this explainer. In a real convolutional network, nobody chooses the numbers in the kernel. They start random and are learned by backpropagation, exactly like the weights of any other layer. A kernel that detects vertical strokes is something the network discovers, because detecting them helps it tell a 1 from a 0.

• A conv layer holds many kernels at once — 32 or 64 is typical — each producing its own feature map. The layer's output is a stack of them.
• A ReLU follows, clipping negatives to zero. That is the default setting above, which is why one side of each edge disappears.
• Pooling then shrinks each map, and the next conv layer runs over that stack — so later layers combine edges into strokes, and strokes into digit-shaped things.
• The kernel is small and is reused at every position, so a conv layer needs very few weights: a 3 × 3 kernel is 9 numbers, whatever the size of the image.

One footnote for the pedants: what CNNs call a convolution is technically a cross-correlation — a true convolution flips the kernel first. Since the weights are learned anyway, the flip makes no practical difference, and PyTorch's nn.Conv2d does exactly what this page does.

The digits are real MNIST test images, loaded with torchvision.datasets.MNIST and embedded in this page, so it runs entirely in your browser with nothing to install. The convolution is the same operation as torch.nn.functional.conv2d, and was checked against it — the two agree to within float32 rounding error.