Derivatives: The Mathematics of Change

The question calculus answers

"If I nudge the input a tiny bit, how much does the output change?"

That single question is the derivative, and it is the engine of all deep learning. Here's why it matters: a model has parameters (those $w$ 's). Training asks, over and over: if I nudge this weight up a little, does my error go down? The derivative answers instantly, for millions of weights at once.

From slope to derivative

For a straight line $y = 3x + 1$ , the slope is always 3: increase $x$ by 1 and $y$ rises by 3, everywhere.

Curves are trickier — the steepness of $y = x^2$ keeps changing. The trick: zoom in. Any smooth curve looks straight under enough magnification. The slope of that local straight line is the derivative at that point, written $\frac{dy}{dx}$ or $f'(x)$ .

For $f(x) = x^2$ , the derivative is $f'(x) = 2x$ :

At $x = 3$ : slope $= 6$ (climbing steeply)
At $x = 0$ : slope $= 0$ (flat — the bottom of the valley!)
At $x = -3$ : slope $= -6$ (descending steeply)

Slope zero marks the bottom. Remember that; it is the entire premise of model training: we search for the flat spot at the bottom of the error curve.

The rules (you need surprisingly few)

Rule	Statement	Example
Power	$\frac{d}{dx} x^n = n x^{n-1}$	$\frac{d}{dx}x^3 = 3x^2$
Constant multiple	$\frac{d}{dx} c f = c f'$	$\frac{d}{dx}5x^2 = 10x$
Sum	$(f + g)' = f' + g'$	derivatives add up
Exponential	$\frac{d}{dx} e^x = e^x$	(its own slope!)
Logarithm	$\frac{d}{dx} \ln x = \frac{1}{x}$	shows up in log-loss

In practice, deep learning frameworks (PyTorch, TensorFlow) compute all derivatives automatically — it's called autodiff. You will rarely differentiate by hand. But you must understand what the numbers mean: a derivative is a sensitivity dial — how much does the output care about this input?

Minima: where learning stops

Training wants to minimize error. At a minimum the derivative is zero — the curve is flat. So a naive recipe for learning is:

Compute the derivative of the error with respect to each parameter.
If the derivative is positive, the error increases with the parameter → decrease the parameter. If negative → increase it.
Repeat until every derivative is (near) zero.

That recipe has a name — gradient descent — and after one more concept (gradients) you'll fully understand it.

Key takeaways

Derivative = local slope = "how sensitive is the output to this input?"
Slope zero ⇒ flat ⇒ possible minimum: exactly what training searches for.
Frameworks differentiate for you (autodiff); your job is interpreting the result.