Statistics for Model Evaluation

From samples to truth

Statistics answers: what can I conclude about the world from the limited data I have? Machine learning lives and dies by this question — a model only ever sees a sample, yet must perform on the whole population.

Sampling bias is the silent killer. Train a face detector only on well-lit studio photos and it fails in the real world. Survey only landline owners (as a famous 1936 US election poll did) and you predict the wrong winner. The data you have is rarely the data you need — always ask "how was this collected, and who is missing?"

Estimators and their reliability

The sample mean $\bar{x} = \frac{1}{n}\sum x_i$ estimates the true mean. How trustworthy is it? Its uncertainty shrinks like

$\text{standard error} = \frac{\sigma}{\sqrt{n}}$

Note the square root: to get 10× more precision you need 100× more data. This is why "more data" is powerful but has diminishing returns — and why the last few percent of model accuracy are so expensive.

Confidence intervals quantify this: "accuracy = 91% ± 2%" is a professional statement; a bare "91%" measured on 50 examples is noise. When comparing two models, always ask whether the gap exceeds the noise — that's the essence of statistical significance (and of A/B testing, the industry's bread and butter).

Correlation is not causation

Correlation measures how two variables move together, from −1 to +1:

$r = \frac{\mathrm{Cov}(X, Y)}{\sigma_X \sigma_Y}$

Ice-cream sales correlate with drowning deaths (both rise in summer). Correlation detects association, never cause. Machine learning models are correlation machines — spectacular at finding patterns, incapable of knowing which patterns are causal. That's why a model may latch onto hospital tags in X-ray corners instead of actual pathology ("shortcut learning"), and why deploying models to make decisions (not just predictions) requires extra care.

The statistics of honest evaluation

Three habits that separate professionals from beginners — you'll formalize them in the ML path:

1. Never evaluate on training data. Memorization masquerades as skill.
2. Report uncertainty, not just point scores.
3. Beware multiple testing: try 100 model variants and the best one's score is inflated by luck. (This is also how "p-hacking" produces junk science.)

Key takeaways

• ML generalizes from samples; biased samples ⇒ biased models, no matter the algorithm.
• Precision improves only with $\sqrt{n}$ — data has diminishing returns.
• Correlation ≠ causation; models find associations and honest evaluation is a statistical discipline.