Language Before Transformers

The problem

Words become data

Chapter 4 gave us networks that learn their own features. Now give one a sentence. A network cannot read letters directly: first we decide what counts as a unit, assign each unit a vocabulary ID, and turn the IDs into numerical inputs. Early language models often used whole words; a modern tokenizer may use word pieces. The choice changes how long a sequence is and what happens to a word never seen before.

An ID is an address, not a measurement. If cat = 17 and dog = 18, the network must not infer that a dog is one more cat. A one-hot vector avoids that false arithmetic: each word owns one coordinate. But in a 50,000-word vocabulary every vector has 50,000 coordinates, and cat is just as orthogonal to dog as to carburettor.

The first idea

Predict one word

Instead of writing rules for language, ask one measurable question again and again: given the words so far, what comes next? A language model must assign a probability to every candidate. After "the cat sat on the", mat ought to get more probability than moon. Training compares those probabilities with what the text actually did.

The chain rule says that a sentence probability is the product of these next-word probabilities:

The models in this chapter differ mainly in what they keep from that history. The first practical answer was to keep almost none of it. A bigram counts what followed one word; a trigram counts what followed two. Divide a continuation's count by the total count for that context and you have a probability.

A tiny worked example, using the 12-sentence corpus in the lab below. Start from the and score the rest of "the cat sat on the mat". The first step has only one word of context, so it uses a bigram; after that, a trigram:

Multiply them: $\tfrac15\cdot\tfrac13\cdot1\cdot1\cdot\tfrac25=\tfrac{2}{75}\approx0.027$. The certain steps cost nothing; the uncertain ones (which animal? which surface?) carry all the doubt. This toy corpus is far too small for the numbers to mean anything about English.

The obstacle

Where counts fail

Try the cat as a trigram context in the lab. The tiny corpus has a few continuations. Now try the mouse. The corpus contains those words but never a following word: the exact context has zero matches. Add-one smoothing removes the zero probability by adding a small count to every candidate, but with no evidence it gives every word the same guess. More careful n-gram systems back off to shorter contexts and smooth more intelligently; exact-count sparsity still grows rapidly as the context gets longer.

The escape is to let similar words share evidence. Bengio and colleagues' 2003 neural language model learned a short vector for each word while learning next-word probabilities. If kitten and cat are used similarly, their vectors can become close, and the model can generalize from a seen phrase to a related unseen one. Its context window was still fixed, but the representation was no longer an exact-match table.

The representation

Words become nearby points

A learned embedding gives each word a dense vector. Training can place words used in similar contexts near one another, so a model can use evidence from neighbouring parts of the space.

Word2vec made learning such vectors fast at large scale. In CBOW, nearby words predict a centre word. In skip-gram, a centre word predicts nearby words. A later word2vec paper introduced negative sampling, which contrasts real word-context pairs with a few sampled non-pairs rather than calculating a full-vocabulary probability for every pair. GloVe took another route: fit word vectors to global word co-occurrence statistics collected across the corpus.

The lab below trains a miniature skip-gram model with negative sampling while you watch. coffee and tea never appear in the same sentence, but both sit next to hot and cup, so training pulls their vectors in the same direction. Start it a few times from different random positions: the picture rotates and flips, but the neighbourhoods come back.

Sequence memory

Remembering a sequence

An n-gram can see only a fixed suffix. A recurrent neural network (RNN) instead reads one word at a time. It combines the current word vector $x_t$ with its previous hidden state $h_{t-1}$ to produce a new state:

The same weights are reused at every step. The hidden state can, in principle, carry a clue from much earlier in the sentence. It is a learned summary, though, not a lossless transcript. To train the network, unfold the repeated cell into a chain and send error backward through the steps: backpropagation through time. The path from a late prediction back to an early word can be long; gradients multiply along it and often vanish or explode. Chapter 4's gradient problem has returned, now across time.

A better memory

Gates help, but time still passes one step at a time

An LSTM adds a cell state and learned gates that control what old information to keep, what new information to write and what to expose. A GRU is a more compact gated recurrent unit with a similar goal. A gate is a number between 0 and 1 multiplied into an information path; the network learns its values from data. This gives useful information and training gradients a better route through many steps.

Gating improved long-range sequence learning. It did not change the ordering constraint: to compute step 20, an RNN or LSTM still needs the state from step 19.

Translation

Translating with one vector

The next challenge was mapping one sequence to another. English "the blue house" becomes French "la maison bleue": a different order, possibly a different length. In a sequence-to-sequence model, an encoder reads the source and a decoder generates the target one word at a time. Influential 2014 recurrent systems from Cho and colleagues and from Sutskever and colleagues made this pattern concrete.

Early versions passed the decoder only the encoder's final state: one fixed-size vector for the whole source.

For a short sentence, that summary can work. For a long one, a decoder may need a precise source detail that the final state failed to retain. This is the fixed-vector bottleneck. It is a practical information-path problem, not a claim that a vector can never encode a long sentence. The encoder has been asked to put every useful detail into one suitcase before the decoder even knows which detail it will need next.

The breakthrough

Attention looks back

Bahdanau, Cho and Bengio's 2014 solution was to keep the encoder's state at every source position. For each target word, the decoder scores how relevant each source state is, normalizes the scores with softmax, and takes their weighted average. The resulting context is different at each decoding step. When producing bleue, the model can look toward blue; when producing maison, toward house. Switch the translation diagram above to With attention and step through the French words to see this.

1. 1

   Keep

   Preserve one encoder state for every source position.
2. 2

   Align

   At this decoder step, score those states and apply softmax.
3. 3

   Read

   Mix the source states by their weights, then use the result to predict the next target word.

Those weights are learned from data and need not correspond to a clean one-to-one translation dictionary. The essential change is direct, step-specific access to the source. This is encoder-decoder attention, often called cross-attention; Chapter 7's self-attention lets positions within a sequence exchange information with one another.

Why it matters

Why it matters

This chapter is the chain of problems behind the Transformer: exact word counts could not generalize, dense word vectors shared evidence, recurrent state extended context, gates made that state more usable, and attention let a decoder retrieve source details rather than trusting one compressed final state.

For a researcher, each step is an example of an architecture changing the path information must travel. For an engineer, the same story explains why model input units, vocabulary design, context length and sequential computation are real system choices rather than cosmetic details.

What came next

To Transformers

Attention had fixed the single-summary bottleneck, but the encoder and decoder were still recurrent. Information within each side still moved through a chain of hidden states, and each step waited for the previous one. The next proposal removed the recurrence and let every position look directly at other positions.