Capstone: The End-to-End ML Workflow

The project lifecycle

Courses teach algorithms; jobs demand projects. Let's walk the full arc with a realistic scenario — predicting customer churn for a subscription service — stitching together every lesson in this path.

1. Frame the problem

Business goal: reduce churn. ML framing: supervised binary classification — for each customer, predict the probability of cancelling within 90 days. Define the metric with stakeholders: retention offers are cheap and missing a churner is expensive → prioritize recall at acceptable precision; track AUC for model comparison. Also set the baseline to beat: the current heuristic ("flag customers inactive 30 days") or a trivial model. If you can't beat the baseline, ship the baseline.

2. Get and audit the data

Assemble features from usage logs, billing, support tickets. Immediately split off the test set and lock it away. Audit what remains: missing values, duplicates, class balance (churners are ~8% — imbalanced!), and above all leakage — the column "sent_cancellation_email" predicts churn perfectly and must go, as must any field written after the prediction moment. Split time-aware: train on older customers, validate on recent ones.

3. Baseline model, then iterate

Start simple: logistic regression on obvious features. It scores AUC 0.71 — already better than the heuristic. Now iterate the loop: engineer features (usage trend last 30 days, support-ticket sentiment, tenure), try gradient-boosted trees (AUC 0.79), tune hyperparameters on the validation set, check learning curves — the gap says a bit overfit → regularize, gather more history. Each experiment is logged: data version, features, params, score. Reproducibility is professionalism.

4. Evaluate like a skeptic

Beyond the headline AUC: inspect the confusion matrix at the chosen threshold; slice performance by segment (new vs. veteran customers, regions, plans) — a model great on average can be terrible on a subgroup, which is both a business and a fairness problem. Sanity-check feature importances: does the model lean on sensible signals or on artifacts? Finally, unlock the test set, measure once, report with uncertainty: AUC 0.78 ± 0.02.

5. Ship and watch it

Deployment turns a model into a service: an API that scores customers nightly. But the world drifts — pricing changes, competitors appear, behavior shifts. Data drift (inputs change) and concept drift (the input→output relationship changes) silently rot models. Monitor input distributions, prediction rates, and realized outcomes; retrain on schedule or trigger. A model without monitoring is a time bomb with a KPI attached.

And the loop closes: monitoring reveals new failure modes → new features → new experiments. ML systems are gardens, not statues.

Where to go next

You can now frame problems, prepare data, train and honestly evaluate classic models. Two directions: build a portfolio project end-to-end on a public dataset (the single best way to consolidate), and continue to the Deep Learning path — where models learn their own features and the modern AI era begins.

Key takeaways

• Real ML = framing → data discipline → baseline → iterate → skeptical evaluation → deploy → monitor.
• Baselines and leakage checks save more projects than clever algorithms do.
• Models decay; monitoring and retraining are part of the job, not an afterthought.