Gradient Descent: Learning as Descent

The loss landscape

Take everything so far and assemble it. A model with parameters $\mathbf{w}$ makes predictions; a loss function $L(\mathbf{w})$ measures how wrong they are on the training data — for example mean squared error:

$L(\mathbf{w}) = \frac{1}{n} \sum_{i=1}^{n} \left( \hat{y}_i - y_i \right)^2$

Imagine $L$ as a landscape: one axis per parameter, altitude = loss. A two-parameter model gives rolling hills in 3-D; GPT-class models give a landscape in billions of dimensions. Training = finding a low point in this landscape.

The algorithm

Gradient descent, in full:

1. Start with random parameters $\mathbf{w}$.
2. Compute the loss $L(\mathbf{w})$ on training data (forward pass).
3. Compute the gradient $\nabla L$ — the uphill compass (backward pass).
4. Step downhill: $\mathbf{w} \leftarrow \mathbf{w} - \eta \nabla L$.
5. Repeat thousands of times until the loss stops improving.

Every headline AI system — image recognition, AlphaGo's networks, ChatGPT — was trained with some variant of these five steps.

The learning rate: the most important dial

$\eta$ controls step size, and tuning it is the classic first lesson of practical ML:

• Too small → training crawls; you wait days for nothing.
• Too large → you leap across valleys, loss oscillates or explodes.
• Common practice → start moderately large, decay the rate over time (big strides early, careful steps near the bottom).

Variants that make it practical

Stochastic gradient descent (SGD). Computing the exact gradient over millions of examples per step is wasteful. Instead, estimate it from a random mini-batch (say 128 examples). The estimate is noisy, but you take thousands of cheap steps instead of few expensive ones — and the noise even helps escape bad regions.

Momentum. Average recent gradients so the update builds speed along consistent directions and damps zigzagging — a heavy ball rolling downhill rather than a cautious hiker.

Adam. The default optimizer in deep learning: combines momentum with a per-parameter adaptive learning rate. When in doubt, people reach for Adam.

What can go wrong

• Local minima — valleys that aren't the lowest point. In practice, in very high dimensions, most such traps are actually saddle points (minimum in some directions, maximum in others), and noisy SGD slides off them.
• Plateaus — vast flat regions where gradients vanish and progress stalls.
• Divergence — a too-large learning rate sends the loss to infinity. Watch the loss curve!

The remarkable empirical fact of deep learning: despite a wildly non-convex landscape, simple gradient descent finds excellent solutions. Understanding why is still partly an open research question.

Key takeaways

• Training = descending a loss landscape; five steps, repeated.
• The learning rate is the most sensitive hyperparameter; mini-batches (SGD) make descent affordable; Adam is the pragmatic default.
• Loss curves are your window into training health — learn to read them.