Probability and Distributions

Must knowKnow well20 minDifficulty

A probability distribution assigns a likelihood to every possible outcome — and nearly every modern model's output is a distribution rather than a single answer.

The problem

The world is uncertain: the next word, a label, a measurement. A model that outputs one hard answer can't say how sure it is.

The solution

Describe uncertainty with a distribution: non-negative numbers over all possible outcomes that sum (or integrate) to 1.

The consequence

Models can be trained by making the correct outcome likely, and their outputs can be sampled, compared and calibrated.

Intuition

Before a die is rolled, you can't say which face will come up, but you can say something precise: each face has probability 1/6. That list — six numbers, each ≥ 0, summing to 1 — is a distribution. The die's outcome is a random variable.

The two you'll meet most

Categorical — a probability for each of $K$ discrete options:

P(X = k) = p_k, \qquad p_k \ge 0, \qquad \sum_{k=1}^{K} p_k = 1

A classifier over 10 digits outputs one. A language model outputs one over its whole vocabulary — e.g. 50,257 probabilities for GPT-2 — every time it predicts a token.

Gaussian (normal) — a bell curve over real numbers, described by a mean $\mu$ (centre) and standard deviation $\sigma$ (spread). Noise in measurements, the random initial values of network weights, and the noise added in diffusion models are usually Gaussian.

Where it appears in AI

Prediction as a distribution: a model says "72% cat, 25% dog, 3% car", not just "cat". Softmax is how raw scores become that distribution.
Generation as sampling: an LLM picks its next token by sampling from its predicted distribution.
Training as matching distributions: the loss functions in this chapter all measure how far the model's distribution is from the data's.

What to remember

A distribution: non-negative values over all outcomes, summing to 1.
Random variable = a quantity whose value is uncertain, described by a distribution.
Categorical distribution: a probability per class or token — what classifiers and LLMs output.
Gaussian (normal): the bell curve, used for noise and weight initialization.