Skip to content
Road to Intelligence

Concept · Chapter 2: The Math Toolkit

Cross-Entropy Loss

Must knowImplement30 minDifficulty

Cross-entropy loss is the negative log of the probability a model assigned to the correct answer — near zero when the model is confidently right, large when it is confidently wrong.

The problem

A classifier or language model outputs a probability distribution; we need a loss that rewards putting high probability on the right answer and is smooth enough to train with gradients.

The solution

Take −log of the probability given to the true class or token, averaged over examples. Minimizing it is the same as maximizing the likelihood of the data.

The consequence

It is the training objective of essentially every classifier and every language model; its average per token, exponentiated, is perplexity.

You should understand first

  1. Probability and Distributions
  2. Entropy
  3. Softmax
  4. Loss Functions
  5. Cross-Entropy Loss

Intuition

A language model reads "The cat sat on the" and assigns probabilities to every possible next word. The true next word was "mat". How good was that prediction? Look only at the probability given to "mat": the higher, the better. Cross-entropy turns that probability into a cost with −log⁡-\log.

Tiny numeric example (natural log)

  1. Confident and right

    p(mat) = 0.9 → loss =−ln⁡0.9≈0.11= -\ln 0.9 \approx 0.11

  2. Hedging

    p(mat) = 0.25 → loss =−ln⁡0.25≈1.39= -\ln 0.25 \approx 1.39 — exactly the loss of a uniform guess over 4 words.

  3. Confident and wrong

    p(mat) = 0.01 → loss =−ln⁡0.01≈4.61= -\ln 0.01 \approx 4.61. The log makes near-zero probabilities very expensive.

The equation

H(p,q)=−∑xp(x)log⁡q(x)→  p one-hot  L=−log⁡q(ytrue)H(p, q) = -\sum_{x} p(x) \log q(x) \quad \xrightarrow{\;p \text{ one-hot}\;} \quad \mathcal{L} = -\log q(y_{\text{true}}) LLM=−1T∑t=1Tlog⁡qθ(xt∣x<t)\mathcal{L}_{\text{LM}} = -\frac{1}{T} \sum_{t=1}^{T} \log q_\theta(x_t \mid x_{<t})

Try it

Try it · toy model

Confident and Wrong

Adjust a model's scores for the next word and watch cross-entropy loss and perplexity respond — the exact quantity language models are trained to minimize.

Know well6 min

Where it appears in AI

  • LLM pretraining is literally "minimize next-token cross-entropy over trillions of tokens".
  • Sanity check: a fresh model with vocabulary VV guesses roughly uniformly, so its initial loss is about ln⁡V\ln V — about 10.8 for a 50,000-token vocabulary. If your first loss is far from that, something is wrong.
  • Cross-entropy == entropy ++ KL divergence, and its exponential is perplexity.

Why should I care?

As a researcher

LLM pretraining loss curves, scaling laws and perplexity comparisons are all cross-entropy; its link to KL divergence also underlies distillation and preference optimization.

As an engineer

It's the number you watch during training. Knowing that ln(vocab size) is the loss of a uniform guess tells you instantly whether a run is learning at all.

Modern systems that depend on it

  • LLM pretraining
  • Every classifier
  • Perplexity
  • Scaling laws
  • Knowledge distillation

Historical context

Before

Squared error was common for classification in early neural networks; information theory (Shannon, 1948) provided the underlying concepts.

After

Became the standard loss for softmax outputs; next-token cross-entropy at scale is the pretraining objective of modern LLMs.

Used today

Every pretraining step of every LLM minimizes the average cross-entropy of the next token.

What to remember

  • Loss = −log p(correct). p = 1 → 0; p → 0 → ∞.
  • Confidently wrong is punished hardest.
  • General form: H(p, q) = −Σ p log q (target p, model q); with a one-hot target it reduces to −log q(correct).
  • Minimizing cross-entropy = maximizing likelihood.
  • Uniform guess over V options gives loss ln V — a useful sanity check.

Key papers

Important

A Mathematical Theory of Communication

C. E. Shannon · 1948 · Bell System Technical Journal

Founded information theory: it defined entropy as a measure of uncertainty and showed how much any message can be compressed. Cross-entropy loss and perplexity come straight from here.

How to read it: Don't read it cover to cover. Part I (sections 1–7) contains entropy and the famous 'series of approximations to English' — a 1948 language model.

~2 h readdoi:10.1002/j.1538-7305.1948.tb01338.x✓ verified 2026-09-26

Watch