Skip to content
Road to Intelligence

Concept · Chapter 6: Language Before Transformers

N-Gram Models

Must knowKnow well16 minDifficulty

An n-gram model predicts the next word by counting what followed the previous n−1 words in a corpus.

The problem

A language model needs conditional probabilities, but full sentences have too many possible histories to count.

The solution

Keep only a short suffix of context and estimate continuation probabilities from observed counts.

The consequence

The model is simple and inspectable, yet sparse contexts produce zero counts and a fixed window loses older information.

You should understand first

  1. Text as Data
  2. Probability and Distributions
  3. Conditional Probability and Bayes' Theorem
  4. Probability of Sequences
  5. Language Modeling
  6. N-Gram Models

Counts into probabilities

If "the cat" is followed by sat twice and slept once, a trigram estimate gives P(sat∣the cat)=2/3P(\text{sat}\mid\text{the cat})=2/3. It is an honest summary of the data. It also gives zero to every continuation absent from that sample, including plausible ones.

Add-one smoothing adds one imaginary count to each vocabulary word. With vocabulary size VV and context count NN, the estimate becomes

P(w∣c)=count⁡(c,w)+1N+V.P(w\mid c)=\frac{\operatorname{count}(c,w)+1}{N+V}.

This prevents zeros. If the context never occurred, however, all words get 1/V1/V: a uniform guess. Better classical models combine different context lengths and use more careful smoothing, but they still cannot share evidence between words merely because those words mean similar things.

The next step was to give words learned numerical representations so an unseen phrase could resemble a seen one.

What to remember

  • A trigram uses two previous words; a bigram uses one.
  • Observed continuations are counted and normalized into probabilities.
  • Smoothing removes zero probabilities but cannot invent a useful interpretation of an unseen context.

Watch