Random Variables & Distributions

Random variables: numbers with personalities

A random variable is a quantity whose value is uncertain — tomorrow's temperature, the next word a user types, the error of a model on a fresh example. A distribution describes its personality: which values are likely, which are rare.

Two numbers summarize a distribution:

• Expectation (mean) $\mathbb{E}[X]$ — the long-run average, the "center of mass".
• Variance $\mathrm{Var}(X) = \mathbb{E}[(X - \mathbb{E}[X])^2]$ — how spread out values are. Its square root, the standard deviation $\sigma$, is in the original units.

Same average, different variance = very different situations. Two models can both average 90% accuracy while one swings wildly between 70% and 99% — you usually want the low-variance one.

Distributions you will actually meet

Bernoulli — a single yes/no event with success probability $p$. A binary classifier's output is a Bernoulli parameter.

Binomial — number of successes in $n$ independent yes/no trials.

Uniform — everything equally likely; how weights are sometimes initialized.

Gaussian (normal) — the bell curve:

$p(x) = \frac{1}{\sigma\sqrt{2\pi}} \, e^{-\frac{(x-\mu)^2}{2\sigma^2}}$

Centered at $\mu$, width set by $\sigma$; ~68% of samples within 1σ, ~95% within 2σ, ~99.7% within 3σ.

Why the Gaussian is everywhere

The Central Limit Theorem (CLT): add up many small independent random effects and the sum tends toward a Gaussian — regardless of the individual effects' distributions. Heights, measurement errors, sensor noise: all bell curves, because all are sums of many tiny causes.

In machine learning the Gaussian appears constantly:

• Noise modeling — "data = truth + Gaussian noise" is the default assumption (and secretly the reason we minimize squared error).
• Weight initialization — networks start with small Gaussian random weights.
• Generative models — diffusion models literally start from pure Gaussian noise and sculpt it into images.

A taste of information theory

How surprising is an event? Rare events carry more information: $-\log p$. The average surprise of a distribution is its entropy. When you later meet cross-entropy loss — the loss function of essentially every classifier and language model — it measures how surprised your model is by the true answers. Lower surprise = better model.

Key takeaways

• Distributions describe uncertain quantities; mean = center, variance = spread.
• The CLT makes Gaussians ubiquitous; squared-error loss quietly assumes Gaussian noise.
• Entropy measures average surprise — cross-entropy loss is "how surprised the model is by the truth".