Data & Feature Engineering

The unglamorous 80%

Practitioners joke that ML is 80% data preparation and 20% modeling — and it's barely a joke. A mediocre algorithm on great data beats a brilliant algorithm on garbage data, every time. This lesson is about that 80%.

Real data is messy

Fresh from the wild, data arrives with:

• Missing values — a sensor died; a user skipped a form field. Options: drop rows (wasteful), fill with the mean/median (imputation), or add a "was-missing" indicator feature (often surprisingly informative).
• Outliers — a typo lists a house at $1 (or 100 rooms). Detect via statistics (e.g. beyond 3σ, from the math path), then investigate — some outliers are errors, others are the most interesting points in the dataset (fraud!).
• Duplicates and leakage — the same row twice, or worse: a feature that secretly contains the answer (e.g. "refund_issued" when predicting fraud). Data leakage produces models with dazzling test scores that collapse in production. It is the most common serious mistake in applied ML.

Turning reality into numbers

Models eat vectors (math path!), so everything must become numbers:

• Categorical features ("red", "green", "blue") → one-hot encoding: three 0/1 columns. Never encode as 1, 2, 3 — that invents a fake ordering (is blue 3× red?).
• Text → counts of words, or learned embeddings (dense vectors capturing meaning — you'll meet them in the DL path).
• Dates → useful pieces: day-of-week, month, is-holiday, days-since-signup.

Scaling: putting features on equal footing

If size ranges 30–400 m² and rooms range 1–8, distance-based algorithms will be dominated by size purely because its numbers are bigger. Two standard fixes:

$\text{standardization: } x' = \frac{x - \mu}{\sigma} \qquad \text{min-max: } x' = \frac{x - x_{\min}}{x_{\max} - x_{\min}}$

Rule of thumb: standardize by default; models based on trees don't care, gradient-based and distance-based models care a lot.

Critical subtlety: compute the scaling parameters ($\mu$, $\sigma$) on the training set only, then apply them to test data. Computing them over all data is leakage — a tiny crack that inflates your scores.

Feature engineering: injecting human insight

Creating informative features from raw ones is where domain knowledge pays off. Predicting taxi fares? Raw GPS coordinates are weak; distance between pickup and dropoff is gold. Classic tricks: ratios (price per m²), aggregates (user's average order), interactions (size × location). Deep learning automates much of this for images/text, but for tabular business data, thoughtful features still win competitions.

Key takeaways

• Data quality dominates algorithm choice; leakage is the classic silent disaster.
• Everything becomes numbers: one-hot for categories, embeddings for text, decomposition for dates.
• Scale features (fit scalers on training data only!); engineer features that encode domain insight.