Skip to content
Road to Intelligence

Concept · Chapter 2: The Math Toolkit

Sampling and Uncertainty

Must knowUnderstand20 minDifficulty

Every measured number — an accuracy, a benchmark score, a loss — is computed from a sample, so it carries uncertainty, and a difference smaller than that uncertainty isn't evidence of anything.

The problem

A model's accuracy on a finite test set is only an estimate of its true accuracy; papers and dashboards often report single numbers as if they were exact.

The solution

Quantify the spread: standard errors, confidence intervals, and results over several random seeds or resamples.

The consequence

You can tell whether 'model B beats model A by 1 point' is real — one of the most important skills for reading papers and running experiments.

You should understand first

  1. Probability and Distributions
  2. Expected Value and Variance
  3. Sampling and Uncertainty

Intuition

Poll 100 random people and 52% prefer tea; poll another 100 and you might get 47%. The true preference hasn't changed — your sample has. Evaluating a model on a test set is exactly the same: the test set is a sample of possible inputs.

Tiny numeric example

A model with true accuracy p=0.8p = 0.8 is evaluated on n=100n = 100 examples:

SE=p(1−p)n=0.8×0.2100=0.04\text{SE} = \sqrt{\frac{p(1-p)}{n}} = \sqrt{\frac{0.8 \times 0.2}{100}} = 0.04

A rough 95% interval is ±2×0.04=±8\pm 2 \times 0.04 = \pm 8 percentage points. The same model could plausibly score anywhere from 72% to 88%. With n=10,000n = 10{,}000 the interval shrinks to about ±0.8\pm 0.8 points.

Try it

Try it · toy model

How Sure Is That Accuracy?

Evaluate the same model on hundreds of random test sets and watch the measured accuracy wobble — why small benchmarks can't separate close models.

Understand5 min

Where it appears in AI

  • Reading papers: a table where methods differ by 0.3 points on a 500-example benchmark, with no error bars, tells you very little. Ask: how big is the test set? How many seeds?
  • Seeds: different random initializations and data orders give different results even with identical code. Serious papers report means and spreads over several runs.
  • Your own experiments: before celebrating an improvement, check it against run-to-run noise. Chapter 3 extends this into train/validation/test splits and generalization; the Epilogue returns to it for reading research.

What to remember

  • A score from n examples has standard error ≈ √(p(1 − p)/n).
  • Rough 95% interval: ± 2 standard errors.
  • 80% accuracy on 100 examples → roughly ±8 points; on 10,000 → roughly ±0.8.
  • Training randomness (seeds) adds more variance: compare several runs, not one.

Watch