Logistic Regression & Classification

Classification: the workhorse task

Spam or not? Fraudulent or legit? Which digit is this? Predicting categories is classification, and despite its name, logistic regression is the fundamental classification algorithm.

The trick: predict a probability

A raw linear model $\mathbf{w}\cdot\mathbf{x} + b$ outputs any number from $-\infty$ to $+\infty$ — useless as a probability. Squash it through the sigmoid function:

$\sigma(z) = \frac{1}{1 + e^{-z}}$

Sigmoid maps any number into $(0, 1)$: large positive → near 1, large negative → near 0, zero → exactly 0.5. So the model becomes

$P(\text{spam} \mid \mathbf{x}) = \sigma(\mathbf{w} \cdot \mathbf{x} + b)$

Output 0.97 reads "97% sure this is spam". To act, apply a threshold — typically 0.5, but a fraud system may flag at 0.2 (cheap to review, expensive to miss) while a system that auto-deletes email should demand 0.99. Choosing thresholds is a business decision, not a math one.

The right loss: cross-entropy

Squared error works poorly for probabilities. Instead we use cross-entropy (log loss) — from the information theory teaser in the math path:

$L = -\frac{1}{n} \sum_{i=1}^{n} \Big[ y_i \log \hat{p}_i + (1 - y_i) \log(1 - \hat{p}_i) \Big]$

Decode it: for a spam example ($y_i = 1$), only $\log \hat{p}_i$ survives — the loss is small if $\hat{p}_i$ is near 1 and explodes toward infinity as $\hat{p}_i \to 0$. Confidently wrong predictions are punished brutally; that steep gradient is exactly what drives fast learning. The same loss, unchanged, trains today's language models (predicting the next word = classification over the vocabulary).

Decision boundaries

Geometrically, logistic regression draws a line (in 2-D; a hyperplane in general) through feature space: probability 0.5 on the line, rising on one side, falling on the other. Linear boundaries are a real limitation — no straight line separates "points inside a circle" from points outside. Fixes: engineer features (add $x^2$ terms), or graduate to trees and neural networks (coming next).

More than two classes

For digits 0–9, generalize sigmoid to softmax, which turns 10 raw scores into 10 probabilities summing to 1:

$P(\text{class } k) = \frac{e^{z_k}}{\sum_j e^{z_j}}$

Softmax + cross-entropy is the standard final layer of virtually every neural classifier ever deployed.

Key takeaways

• Logistic regression = linear model + sigmoid → calibrated probabilities; thresholds turn them into decisions.
• Cross-entropy punishes confident wrongness hard — the universal classification loss, up to and including LLMs.
• The boundary is linear; when that's too rigid, richer models take over.