Regularization

Fighting overfitting by handicapping the model

Overfit models are typically too free — with enough flexible parameters they contort through every training point. Regularization deliberately restricts that freedom, accepting slightly worse training fit in exchange for much better generalization. It is the practical embodiment of Occam's razor: prefer the simplest explanation that fits.

Penalty methods: taxing large weights

Wild, overfit curves need large weights to produce their wiggles. So add a tax on weight size to the loss:

$L_{\text{ridge}} = \text{MSE} + \lambda \sum_j w_j^2 \qquad L_{\text{lasso}} = \text{MSE} + \lambda \sum_j |w_j|$

• Ridge (L2) — the squared-weights tax. Shrinks all weights smoothly toward zero; correlated features share credit. (You met this exact formula in the math capstone!)
• Lasso (L1) — the absolute-value tax. Geometry makes it push weights to exactly zero, performing automatic feature selection: with 10,000 candidate features, lasso might keep 40 and zero out the rest — a model you can actually explain.
• Elastic net — a blend of both.

The knob $\lambda$ sets tax severity: $\lambda = 0$ recovers plain regression (overfit risk); huge $\lambda$ crushes all weights toward zero (underfit). $\lambda$ is a hyperparameter — tuned on the validation set, exactly as last lesson prescribed. One practical note: penalties compare weight sizes across features, so features must be scaled first.

Regularization beyond penalties

The principle "constrain the model" wears many costumes:

• Early stopping — watch validation error during iterative training and stop when it starts rising, even as training error keeps falling. Free, and universally used in deep learning.
• Dropout — randomly silence a fraction of a neural network's units during each training step, so no unit can over-rely on others (details in the DL path).
• Data augmentation — synthetically enlarge the dataset (flip/rotate images, paraphrase text); harder to memorize, more signal to learn.
• Smaller model / fewer features — the bluntest instrument, often the right one.

A worked mental example

Fit a degree-15 polynomial to 20 noisy points: unregularized, it swings violently between points (weights in the thousands), training error ≈ 0, test error terrible. Add ridge with modest $\lambda$: the curve relaxes into a gentle shape close to the true trend; training error rises slightly, test error drops dramatically. That trade — pay a little bias, save a lot of variance — is regularization in one sentence.

Key takeaways

• Regularization trades a bit of training fit for a lot of generalization.
• Ridge shrinks weights; lasso zeroes them (feature selection); $\lambda$ is tuned on validation data.
• Early stopping, dropout and augmentation are the same philosophy in different clothes.