Skip to content
Road to Intelligence

Concept · Chapter 2: The Math Toolkit

Entropy

Must knowKnow well20 minDifficulty

Entropy measures how uncertain a distribution is — the average surprise of its outcomes — and it sets the lower limit on how compactly you can encode them.

The problem

We need a single number for 'how unpredictable is this?' that behaves sensibly: zero for a certain outcome, largest when everything is equally likely.

The solution

Define the surprise of an outcome as −log p (rare events are more surprising) and take its expected value over the distribution.

The consequence

It becomes the reference point for cross-entropy loss and perplexity, and a way to talk about a model's uncertainty in its predictions.

You should understand first

  1. Probability and Distributions
  2. Entropy

Intuition: surprise

"The sun rose this morning" tells you almost nothing — it was nearly certain. "It snowed in the Sahara" tells you a lot. Information is surprise, and surprise is higher for less likely events: −log⁡p-\log p. Entropy is the average surprise you expect from a source.

Tiny numeric example (in bits)

  1. Fair coin

    H=−(0.5log⁡20.5+0.5log⁡20.5)=1H = -(0.5 \log_2 0.5 + 0.5 \log_2 0.5) = 1 bit. Maximally uncertain for two outcomes.

  2. Biased coin, 90/10

    H=−(0.9log⁡20.9+0.1log⁡20.1)≈0.137+0.332=0.47H = -(0.9 \log_2 0.9 + 0.1 \log_2 0.1) \approx 0.137 + 0.332 = 0.47 bits. More predictable, lower entropy.

  3. Four equally likely words

    H=log⁡24=2H = \log_2 4 = 2 bits: you need 2 yes/no questions to identify the word.

The equation

H(p)=−∑xp(x)log⁡p(x)=Ex∼p[−log⁡p(x)]H(p) = -\sum_{x} p(x) \log p(x) = \mathbb{E}_{x \sim p}\big[-\log p(x)\big]

Where it appears in AI

  • Model uncertainty: the entropy of an LLM's next-token distribution is low when it's confident, high when many continuations are plausible.
  • Loss functions: cross-entropy is the surprise of the true outcome under the model's predictions — entropy's close cousin.
  • Decision trees (Chapter 3) choose splits that reduce entropy the most ("information gain").

What to remember

  • Surprise of an outcome = −log p. Certain events: 0 surprise.
  • Entropy H(p) = −Σ p log p = average surprise.
  • Uniform over K outcomes → maximum entropy, log K.
  • Base-2 log → bits; natural log → nats (what ML libraries use).

Key papers

Important

A Mathematical Theory of Communication

C. E. Shannon · 1948 · Bell System Technical Journal

Founded information theory: it defined entropy as a measure of uncertainty and showed how much any message can be compressed. Cross-entropy loss and perplexity come straight from here.

How to read it: Don't read it cover to cover. Part I (sections 1–7) contains entropy and the famous 'series of approximations to English' — a 1948 language model.

~2 h readdoi:10.1002/j.1538-7305.1948.tb01338.x✓ verified 2026-09-26

Watch