Overfitting, Underfitting & Validation

The student who memorized the textbook

Imagine a student who memorizes every practice exam answer verbatim — perfect scores on practice, disaster on the real exam. That is overfitting: the model learned the training data's noise and quirks instead of the underlying pattern. Its mirror image is underfitting — a model too simple to capture the pattern at all (fitting a straight line to a clearly curved relationship).

Generalization — performance on data the model has never seen — is the only thing that matters. Training accuracy is trivially inflatable; a lookup table achieves 100%.

The bias-variance trade-off

Expected error decomposes into three parts:

$\text{Error} = \underbrace{\text{Bias}^2}_{\text{too simple}} + \underbrace{\text{Variance}}_{\text{too sensitive}} + \underbrace{\text{Noise}}_{\text{irreducible}}$

• High bias (underfit): systematically wrong everywhere; more data won't help — the model can't express the truth.
• High variance (overfit): exquisitely tuned to this training set; retrain on slightly different data and predictions change wildly.

Model complexity slides you along the trade-off: too simple → bias dominates; too complex → variance dominates; the sweet spot is in between. Diagnosis is simple and you'll use it forever: training error high? → bias. Training error low but validation error high? → variance.

The three-way split

The professional discipline that keeps you honest:

Why three, not two? Because choosing among 50 model variants based on validation scores slowly overfits the validation set itself (multiple testing — statistics path). The test set stays in a vault until the very end and is used once. Peeking at it during development is the cardinal sin of ML; scores reported after test-set peeking are fiction.

Cross-validation stretches small datasets: split into $k$ folds, train $k$ times each holding out a different fold, average the scores. Slower, but every point serves as validation once — and you get an uncertainty estimate for free.

Time-series caveat: with temporal data, always split by time (train on the past, validate on the future). Random splits let the model "see the future" — a classic leakage disaster in forecasting and finance.

Learning curves: your diagnostic X-ray

Plot training and validation error vs. training-set size. Curves converging at high error → bias problem (get a richer model). Large persistent gap → variance problem (get more data, or regularize — next lesson).

Key takeaways

• Only generalization counts; training scores are propaganda.
• Diagnose with the bias/variance lens: underfit vs. overfit have opposite cures.
• Guard the three-way split; touch the test set exactly once, ever.