Linear Regression

The "hello world" that never retires

Linear regression predicts a number as a weighted sum of features:

$\hat{y} = w_1 x_1 + w_2 x_2 + \cdots + w_n x_n + b = \mathbf{w} \cdot \mathbf{x} + b$

House price = (weight for size)·size + (weight for age)·age + … + base price. It's over 200 years old (Gauss used it to locate an asteroid), yet it remains the most deployed model in the world: fast, stable, and every weight is readable — $w_{\text{size}} = 3200$ literally means "each extra m² adds $3,200, all else equal".

What does "best line" mean?

Define wrongness, then minimize it. The standard choice is mean squared error (MSE):

$L(\mathbf{w}, b) = \frac{1}{n} \sum_{i=1}^{n} \left( \hat{y}_i - y_i \right)^2$

Why squared? It punishes big misses disproportionately, it's smooth (differentiable everywhere — the calculus path explains why that matters), and it corresponds to assuming Gaussian noise (the statistics connection). Plot MSE against the parameters and you get a perfect bowl — convex, one global minimum, no traps.

Finding the minimum

Two routes to the bottom of the bowl:

The normal equation — solve directly with linear algebra: $\mathbf{w} = (X^TX)^{-1}X^T\mathbf{y}$ . Exact, elegant, but slow for many features.
Gradient descent — the iterative descent from the math path. For MSE the gradient has a beautifully interpretable form: each weight's update is proportional to error × its feature value. Wrong by a lot on big-featured examples → big correction.

Route 2 is the same procedure that trains trillion-parameter language models. Linear regression is where you watch it work in a landscape simple enough to fully understand.

Beyond straight lines (without leaving linear-land)

"Linear" refers to linearity in the weights — you can feed in transformed features: $x^2$ , $\log x$ , $x_1 x_2$ . Fitting $\hat{y} = w_1 x + w_2 x^2 + b$ gives a curve, trained exactly like before. This is polynomial regression, and it raises a fateful question: higher-degree polynomials fit training data ever better… but wiggle absurdly between points. Sound dangerous? It is — that's overfitting, and it gets its own lesson soon.

Reading the diagnostics

After fitting, professionals check:

R² ("R-squared") — fraction of the label's variance the model explains (1.0 = perfect, 0 = no better than predicting the mean).
Residual plots — plot errors vs. predictions; visible patterns mean the model is missing structure (maybe the relationship isn't linear).

Key takeaways

Linear regression: $\hat{y} = \mathbf{w}\cdot\mathbf{x} + b$ , trained by minimizing MSE — a convex bowl with one minimum.
Its weights are directly interpretable — a huge practical virtue.
Feature transforms give curves; flexibility invites overfitting (next chapters).