Metrics: Measuring What Matters

The accuracy trap

A disease affects 1% of patients. A "model" that always says healthy scores 99% accuracy — while missing every single sick patient. On imbalanced data (fraud, disease, defects — i.e. most interesting problems), accuracy is a vanity metric. You need sharper tools.

The confusion matrix

Every binary classifier's outcomes fit a 2×2 grid:

The two error types are rarely equal in cost: a false negative in cancer screening can be fatal; a false positive means a stressful follow-up test. Name your costlier error before choosing any metric.

Precision and recall

$\text{Precision} = \frac{TP}{TP + FP} \qquad \text{Recall} = \frac{TP}{TP + FN}$

• Precision: of everything flagged positive, how much truly was? (How trustworthy are alarms?)
• Recall: of all true positives out there, how many did we catch? (How few slip through?)

They fight: flag more aggressively and recall rises while precision falls. The threshold from the logistic-regression lesson is the lever. Cancer screening wants high recall (catch everyone, tolerate false alarms); auto-blocking emails wants high precision (never eat a real email). The F1 score — the harmonic mean $2PR/(P{+}R)$ — compresses both into one number when you must rank models, punishing whichever is worse.

Threshold-free evaluation: ROC and AUC

Sweep the threshold from 0 to 1 and plot true-positive rate vs. false-positive rate: the ROC curve. Its area, AUC, summarizes ranking quality: AUC = 0.5 is a coin flip; 1.0 is perfection. Interpretation: AUC is the probability a random positive example scores higher than a random negative one. For heavily imbalanced data, the precision-recall curve is often more informative than ROC.

Regression metrics, briefly

• MAE — mean absolute error: robust, in original units, "typical miss".
• RMSE — root mean squared error: punishes large misses; the default companion of MSE training.
• R² — variance explained (from the regression lesson).

The metric IS the product decision

Teams fail not by computing metrics wrong but by optimizing the wrong one. A recommender maximizing clicks learns clickbait. A support-ticket classifier maximizing accuracy ignores the rare-but-critical "legal complaint" class. Choosing the metric is choosing what the system will become — it deserves as much thought as any architecture choice.

Key takeaways

• Accuracy lies on imbalanced data; start from the confusion matrix and the cost of each error type.
• Precision = alarm trustworthiness; recall = catch rate; thresholds trade one for the other; F1/AUC summarize.
• Pick the metric that mirrors real-world costs — it silently defines your product.