Decision Trees, Random Forests & Boosting

Decision trees: 20 questions, automated

A decision tree classifies by asking a sequence of yes/no questions:

Income > 50k?
├── no  → Renting long? ── yes → DENY  / no → REVIEW
└── yes → Existing debt high? ── yes → REVIEW / no → APPROVE

Training greedily picks, at each node, the question that best purifies the split — measured by Gini impurity or entropy (the "surprise" measure from the math path). Recurse until leaves are pure or a depth limit hits.

Trees are wonderful: no feature scaling needed, they handle numbers and categories, capture nonlinear patterns and interactions naturally, and you can read them — priceless in regulated industries. But a single deep tree is a memorization machine: grown unchecked, it carves a leaf for every quirk of the training set and shatters on new data. High variance, in the statistical sense: tiny data changes yield a totally different tree.

The ensemble insight: crowds beat experts

Average many diverse, individually-mediocre models and the errors cancel — variance shrinks like $\frac{\sigma}{\sqrt{n}}$ (statistics path!). Two great recipes:

Random forests (bagging). Train hundreds of trees, each on a bootstrap sample of the data and with random feature subsets at each split (forcing diversity); average their votes. Nearly tuning-free, hard to overfit, embarrassingly parallel. The reliable default for tabular data.

Gradient boosting. Train trees sequentially, each one fitting the residual errors of the ensemble so far:

$F_{m}(\mathbf{x}) = F_{m-1}(\mathbf{x}) + \eta \cdot h_m(\mathbf{x})$

where $h_m$ is a small tree trained on the current mistakes and $\eta$ is a learning rate. It is literally gradient descent, but in function space — each tree is one downhill step. Implementations like XGBoost and LightGBM have won more Kaggle tabular competitions than every other method combined, and quietly power credit scoring, ranking and fraud systems across the industry.

Boosting is more powerful than bagging but has more knobs (tree depth, learning rate, number of trees) and can overfit — tune with a validation set.

Deep learning isn't always the answer

A fact that surprises newcomers: on small-to-medium tabular business data, gradient-boosted trees still routinely beat neural networks. Deep learning owns images, audio and text; trees own spreadsheets. Choosing the right tool family for the data type is a mark of professionalism.

Key takeaways

• Trees = learned decision rules: interpretable, scale-free, but high-variance alone.
• Forests average diverse trees (variance ↓); boosting stacks error-correcting trees (bias ↓).
• For tabular data, boosted trees are the industry default — not neural networks.