Backpropagation

The question training must answer

A network has millions of weights and produces a single loss number. To run gradient descent we need $\frac{\partial L}{\partial w}$ for every weight: "if I nudge this one weight, how does the loss respond?" Computing each by trial-and-nudging would take one forward pass per weight — millions of passes per step. Backpropagation gets all of them in roughly the cost of two passes. It is the algorithm that makes deep learning economically possible.

The core idea: sensitivities multiply along chains

You already own the key (math path): the chain rule. A weight deep in the network influences the loss through a chain of intermediate values; the sensitivity is the product of local sensitivities along the chain:

$\frac{\partial L}{\partial w} = \frac{\partial L}{\partial a_3} \cdot \frac{\partial a_3}{\partial a_2} \cdot \frac{\partial a_2}{\partial a_1} \cdot \frac{\partial a_1}{\partial w}$

The efficiency trick: chains overlap. The sensitivity of the loss to layer 2's outputs is shared by every weight in layers 1 and 2. So compute once, working backwards from the loss, layer by layer, caching as you go:

1. Forward pass — run the input through; store each layer's activations.
2. Backward pass — start with $\frac{\partial L}{\partial \hat{y}}$ (easy: differentiate the loss formula), then repeatedly apply the chain rule to push the "error signal" $\delta$ one layer back, reusing the cached activations.
3. Per-weight gradient — beautifully local: $\frac{\partial L}{\partial w_{ij}} = \delta_j \cdot a_i$ — (error at the destination) × (activation at the source). Connections between highly-active sources and high-error destinations get the biggest corrections. Hebb's "neurons that fire together, wire together", but with calculus.
4. Update — gradient descent step; repeat.

Autodiff: backprop as infrastructure

You will never code this by hand. PyTorch/TensorFlow record every operation in a computational graph during the forward pass, then traverse it in reverse, applying each operation's known local derivative. loss.backward() — one line. But when training misbehaves, engineers debug gradients: their magnitudes, their flow, where they die. Understanding backprop is what separates "it doesn't work ¯\(ツ)/¯" from "layer 3's gradients vanish; let's fix the activations".

The disease of depth: unstable gradients

The backward pass multiplies many local derivatives. Multiply fifty numbers each slightly below 1 → the product is ~0: vanishing gradients; early layers receive no signal and never learn. Each slightly above 1 → explosion: NaNs. This single arithmetic fact shaped a decade of research. The standard medicines (details next lesson): ReLU activations (derivative exactly 1 when active), careful weight initialization, normalization layers, and — for very deep nets — skip connections that give gradients a highway past layers.

Key takeaways

• Backprop = chain rule + caching: all million gradients for the price of ~two passes.
• Gradient of a weight = (error at its target neuron) × (activation of its source neuron).
• Deep chains multiply derivatives → vanishing/exploding gradients; most architecture tricks exist to stabilize this flow.

Press Run to execute the code (it runs in your browser, in a sandbox).