Skip to content
Road to Intelligence

Concept · Chapter 2: The Math Toolkit

Perplexity

Must knowKnow well15 minDifficulty

Perplexity is the exponential of the average cross-entropy per token — roughly, the number of options a language model is effectively choosing between at each step.

The problem

Cross-entropy values like 2.3 nats per token are hard to interpret or compare intuitively.

The solution

Exponentiate the average per-token loss to get an 'effective branching factor': a perplexity of 10 means the model is as uncertain as if it were choosing uniformly among 10 tokens.

The consequence

It became the standard intrinsic metric for language models — useful for comparing models on the same data and tokenizer, but not a direct measure of usefulness.

You should understand first

  1. Probability and Distributions
  2. Entropy
  3. Softmax
  4. Loss Functions
  5. Cross-Entropy Loss
  6. Conditional Probability and Bayes' Theorem
  7. Probability of Sequences
  8. Perplexity

Tiny numeric example

  1. Uniform guessing

    A model that spreads probability evenly over 4 words has loss ln⁡4≈1.386\ln 4 \approx 1.386 per token; e1.386=4e^{1.386} = 4. Perplexity 4: "choosing among 4".

  2. A real-looking number

    An average loss of 2.0 nats per token gives perplexity e2.0≈7.4e^{2.0} \approx 7.4.

  3. Small differences matter

    Going from loss 2.0 to 1.9 lowers perplexity from 7.4 to 6.7 — about a 10% reduction in effective choices.

The equation

PPL=exp⁡ ⁣(−1T∑t=1Tlog⁡qθ(xt∣x<t))\text{PPL} = \exp\!\left(-\frac{1}{T} \sum_{t=1}^{T} \log q_\theta(x_t \mid x_{<t})\right)

What to remember

  • Perplexity = exp(average cross-entropy per token) (or 2^loss if the loss is in bits).
  • Uniform guessing over V tokens → perplexity V.
  • Lower is better; 1 would be perfect prediction.
  • Only comparable across models with the same tokenizer and test data.