Skip to content
Road to Intelligence

Concept · Chapter 6: Language Before Transformers

Language Modeling

Must knowKnow well15 minDifficulty

A language model assigns a probability to each possible next text unit given the units before it.

The problem

To generate or score a sentence, a system needs a way to say which continuations are plausible in context.

The solution

Estimate a conditional distribution over the vocabulary at each position, then chain those conditional probabilities across the sequence.

The consequence

The same prediction task can train count tables, neural networks, RNNs and Transformers, making their differences easy to compare.

You should understand first

  1. Text as Data
  2. Probability and Distributions
  3. Conditional Probability and Bayes' Theorem
  4. Probability of Sequences
  5. Language Modeling

One question, repeated

After "the cat sat on the", a model might give mat more probability than moon. It must assign a probability to every possible next word, not merely return its favourite. The model can then continue one step, append a word, and ask again.

The chain rule turns those local answers into a probability for a sequence:

P(w1,…,wT)=∏t=1TP(wt∣w1,…,wt−1).P(w_1,\ldots,w_T)=\prod_{t=1}^{T}P(w_t\mid w_1,\ldots,w_{t-1}).

The challenge is the conditioning phrase on the right. A trigram table sees only the last two words. An RNN compresses the whole prefix into a hidden state. A Transformer can directly compare positions in its available context. All are trying to answer the same next-unit question.

Good next-word predictions require learning useful statistical structure in language Established. The objective alone does not establish that a model has human-like understanding; that requires separate evidence and evaluation.

Why should I care?

As a researcher

The objective stays nearly constant while the context representation changes; that isolates what each architecture contributes.

As an engineer

Next-token probabilities determine autocomplete, generation, perplexity and the loss used to train an LLM.

Modern systems that depend on it

  • n-gram models
  • neural language models
  • RNNs
  • Transformers

Historical context

Before

Earlier language systems often relied on hand-built grammar and task-specific rules.

After

Prediction from data gave a measurable objective; models differed in how they represented and used context.

Used today

Autoregressive LLMs still predict the next token, although the context and model are far larger.

What to remember

  • The output is a full probability distribution, not just one guessed word.
  • P(sentence) is a product of next-unit conditional probabilities by the chain rule.
  • A training objective does not by itself tell you how well a model understands or reasons.

Watch