Probability: Quantifying Uncertainty

Machine learning is applied probability

A model never says "this email IS spam". It says "this email is spam with probability 0.97". Classification outputs, language model word choices, uncertainty estimates — all probabilities. This lesson gives you the grammar of that language.

The basics

A probability is a number between 0 (impossible) and 1 (certain). For a fair die, $P(\text{rolling a 4}) = \frac{1}{6}$. Probabilities over all possible outcomes sum to 1 — a rule that neural networks enforce with their final layer (softmax) so their outputs can be read as probabilities.

Two workhorse rules:

• Sum rule (OR, mutually exclusive): $P(A \text{ or } B) = P(A) + P(B)$
• Product rule (AND, independent): $P(A \text{ and } B) = P(A) \cdot P(B)$

Independence matters: coin flips don't influence each other, but "it rains" and "people carry umbrellas" are not independent. Most interesting real-world events aren't — which brings us to the key concept.

Conditional probability: updating on evidence

$P(A \mid B)$ reads "the probability of $A$ given that $B$ happened". Knowing $B$ changes the odds of $A$:

$P(A \mid B) = \frac{P(A \text{ and } B)}{P(B)}$

Example: $P(\text{spam}) \approx 0.5$ for a random email, but $P(\text{spam} \mid \text{contains "FREE OFFER!!!"}) \approx 0.99$. Evidence updates belief. All of machine learning inference is conditional probability: $P(\text{label} \mid \text{data})$.

Bayes' theorem: reversing the question

Often we know probabilities in one direction but need the other. Doctors know $P(\text{positive test} \mid \text{disease})$; patients want $P(\text{disease} \mid \text{positive test})$. Bayes' theorem flips the conditioning:

$P(A \mid B) = \frac{P(B \mid A) \, P(A)}{P(B)}$

A famous surprise: a disease affects 1 in 1,000 people; a test is 99% accurate. You test positive — how worried should you be? Among 1,000 people, ~1 truly has the disease (and likely tests positive), while ~10 healthy people also test positive (1% of 999). So your chance of actually being sick is roughly $\frac{1}{11} \approx 9\%$, not 99%! The prior (rarity of the disease) matters enormously.

This machinery powers Naive Bayes spam filters, medical diagnosis systems, and the entire Bayesian school of machine learning, where training is viewed as updating beliefs about parameters given data.

Key takeaways

• Probabilities: 0 to 1, sum to 1; models output them, softmax enforces them.
• Conditional probability updates beliefs on evidence — prediction is computing $P(\text{label} \mid \text{data})$.
• Bayes' theorem reverses conditionals and weighs in priors; intuition often gets this badly wrong.