The network sees a raw image of a handwritten digit. Pixel values flow into the first layer, get transformed by weights, pass to the next layer, and eventually produce a score for each possible class (0–9).
At the start of training the weights are random, so the predictions will be garbage — that's fine. The whole point of training is to fix them.
What happens inside each layer?
step 1 weighted sum
z = W · x + b
multiply inputs by weights, add bias
step 2 activation
a = ReLU(z)
add non-linearity so layers aren't redundant
Each layer passes its output as input to the next:
a₁ → Layer 2 → a₂ → … → Y'
This chain of operations — from raw pixels to class scores — is called the forward pass. It's fast, deterministic, and fully defined by the current weights.
Predictions Y' — one score per class
True label: 4
The network outputs a probability for each digit 0–9. Ideally class 4 should score highest. Before training, the scores are essentially random — the network has no idea yet.
Loss function — how wrong are we?
Prediction for class 4
0.12
True target Y
1.00
loss cross-entropy loss
L = −log(0.12) = 2.12
Higher confidence in the wrong answer → higher loss. Perfect prediction → loss = 0.
The loss function compares the network's output to the true label Y. It produces a single number — the loss score — that summarises "how wrong" the network is on this example.
Lower loss = better. Training = minimising this number across all training examples.
Backpropagation computes the gradient of the loss with respect to every weight in the network — using the chain rule of calculus. This tells each weight: "if you increase slightly, does the loss go up or down, and by how much?"
Then gradient descent applies the update: W ← W − α · ∂L/∂W. Every weight gets nudged in the direction that reduces loss.