Gradients & the Chain Rule

Many inputs, one compass

Real models have millions of parameters, so the error is a function of millions of variables: $L(w_1, w_2, \dots, w_n)$. The derivative with respect to one variable while holding the others still is a partial derivative, written $\frac{\partial L}{\partial w_1}$.

Stack all the partial derivatives into a vector and you get the gradient:

$\nabla L = \left[ \frac{\partial L}{\partial w_1},\; \frac{\partial L}{\partial w_2},\; \dots,\; \frac{\partial L}{\partial w_n} \right]$

The gradient has a beautiful geometric meaning: it points in the direction of steepest increase. Picture standing on a foggy hillside where altitude = error. The gradient is a compass needle pointing straight uphill. To reduce error, walk the opposite way:

$w \leftarrow w - \eta \, \nabla L$

This is gradient descent — the update rule that trains essentially every modern AI system. The small number $\eta$ (eta, the learning rate) controls the step size: too large and you overshoot the valley, too small and training takes forever.

The chain rule: derivatives through pipelines

Remember that deep networks are chained functions: $f(g(h(x)))$. How does the final output respond to a nudge deep inside the chain? The chain rule answers: multiply the sensitivities of each link.

$\frac{dy}{dx} = \frac{dy}{du} \cdot \frac{du}{dx}$

An analogy: if your salary grows 2× as fast as your company's revenue, and revenue grows 3× as fast as the market, then your salary grows 2 · 3 = 6× as fast as the market. Sensitivities along a chain multiply.

Concrete example: $y = (3x + 1)^2$. Let $u = 3x+1$ (inner) so $y = u^2$ (outer).

$\frac{dy}{dx} = \underbrace{2u}_{dy/du} \cdot \underbrace{3}_{du/dx} = 6(3x+1)$

Why this is the most important idea in deep learning

A neural network is a chain of thousands of simple operations. To train it we need the gradient of the error with respect to every weight, including those at the very start of the chain. The chain rule lets us compute all of them in one sweep — starting from the error and multiplying sensitivities backwards through the network, layer by layer.

That backward sweep is called backpropagation. When you reach the Deep Learning path, you'll see it in detail — but the math is literally just this lesson, applied systematically. If you understand "sensitivities multiply along a chain", you already understand how ChatGPT was trained.

Key takeaways

• Gradient = vector of all partial derivatives = compass pointing uphill on the error landscape.
• Gradient descent: repeatedly step downhill, $w \leftarrow w - \eta\nabla L$.
• Chain rule: sensitivities along a chain multiply — the engine behind backpropagation.