Transformers

The problem

One word at a time

By 2016, the best language systems — translation above all — were built from recurrent neural networks (Chapter 6). An RNN reads a sentence the way you'd read through a keyhole: one word at a time, carrying a single summary vector forward. Attention had already been bolted on to help the decoder look back at the input, but inside the encoder and decoder, information still crawled from word to word.

That created two problems that grew worse as data and ambitions grew:

• Distance. For the model to connect it to animal six words earlier, that information has to survive six sequential updates of the hidden state — and real dependencies can span hundreds of words.
• Speed. Step 10 can't be computed before step 9. Training can't use the thousands of parallel cores in a GPU across the length of the sequence.

The idea

Let every word look at every word

The Transformer's authors took the attention mechanism — until then an add-on to RNNs — and made it the whole architecture. Every token directly compares itself with every other token and gathers what it needs, in a single step, with all tokens computed at once.

Remove recurrence entirely. Keep attention. Add small per-token neural networks, and a few engineering tricks that make deep stacks trainable. That is the entire design.

How it works

Attention from first principles

Don't start with matrices. Start with a question.

I have a sentence. For each word, which other words should it pay attention to?

You already have every tool needed. Words are vectors (embeddings). The dot product scores how aligned two vectors are. Softmax turns scores into weights that sum to 1. A weighted sum mixes vectors by those weights. Put them together:

1. 1

   Score

   For the word asking the question, compute a similarity score with every word.
2. 2

   Normalize

   Softmax the scores into attention weights.
3. 3

   Mix

   The word's new vector is the weighted average of all word vectors.

Try it — and watch what goes wrong in step 1 for the word it:

Queries, keys and values

Step 1 exposed a flaw: raw similarity asks "who is like me?", but it needs to ask "who could I be referring to?". What a word looks for differs from what it is.

So each token produces three vectors through three learned matrices:

• a query — what I'm looking for,
• a key — what I offer to others,
• a value — the information I pass on if selected.

Scores become query · key; the weighted sum is over values. During training, the model learns $W_Q$, $W_K$ and $W_V$ — it learns what to look for. Two more details complete the mechanism: scores are divided by $\sqrt{d_k}$ before the softmax, and in a GPT-style model a causal mask hides every later token.

Many heads at once

One attention pattern gives each token a single weighted average. Language needs several at once — who did it, to what, what came just before. So the Transformer runs h heads in parallel, each in a smaller subspace with its own $W_Q, W_K, W_V$, then concatenates their outputs and mixes them with one more matrix, $W_O$. With 8 heads of 64 dimensions instead of one of 512, the cost is about the same.

Where is everything?

There's a catch. Attention compares every token with every other using dot products — nothing in that computation knows where a token is. Shuffle the words and each token gets exactly the same scores, just in a different order. "Dog bites man" and "man bites dog" would look alike.

The fix is to inject position. The original Transformer adds a fixed pattern of sine and cosine waves to each embedding; GPT-2 and BERT learn a vector per position; many recent LLMs rotate queries and keys by position (RoPE).

The Transformer block

Attention alone only averages. Each block therefore pairs it with a feed-forward network applied to every token separately — expand to a wider layer, apply a nonlinearity, project back. Around both sublayers sit two engineering necessities: layer normalization to keep numbers in a stable range, and residual connections so each sublayer adds a correction rather than replacing its input.

The output has the same shape as the input, so blocks stack: 6 in the original encoder and decoder, 12 in GPT-2 small, on the order of a hundred in today's largest models. Step through one below — every number is really computed, just with tiny dimensions and untrained weights.

Three families

The original Transformer was an encoder–decoder for translation: an encoder reads the source sentence with full, bidirectional attention; a decoder generates the translation with causal attention plus cross-attention to the encoder. Within two years the halves were being used on their own.

The math

The math in one place

Everything above, as equations. If the chapter made sense, each line should read as a sentence.

Why it matters

Why it matters

It solved the two problems it set out to solve. Any token reaches any other in one step, and the whole sequence is processed in parallel, which suits GPU hardware.

It scaled. The same uniform block could be made wider and stacked deeper, trained on more data, and kept improving. That property — more than any single benchmark result — is why nearly every large model since descends from it. Chapter 8 tells that story.

It generalized beyond language. The block makes no assumptions about text: images cut into patches (Vision Transformers), audio frames (speech models) and mixtures of modalities all run through essentially the same architecture.

Historical context

Attention Is All You Need

In June 2017, eight researchers, most of them at Google Brain and Google Research, posted Attention Is All You Need. It proposed the Transformer for machine translation and reported 28.4 BLEU on WMT 2014 English-to-German — more than 2 BLEU above the previous best results, including ensembles — and a new single-model state of the art of 41.8 on English-to-French, after training for 3.5 days on eight GPUs: a fraction of the cost of the best earlier models.

The ideas it combined were not all new — attention (2014), residual connections (2015), layer normalization (2016) — but the decision to drop recurrence entirely, and the specific, simple way they were assembled, proved decisive. Within eighteen months, BERT (an encoder) and GPT (a decoder) had shown that pretraining a Transformer on unlabelled text and then adapting it beat task-specific models across NLP.