Skip to content
Road to Intelligence

Concept · Chapter 7: Transformers

Causal Masking

Must knowKnow well15 minDifficulty

A causal mask stops each position from attending to later positions, so a model trained to predict the next token can't simply look at it.

The problem

When training on a whole sentence at once, self-attention would let position 3 see the word at position 4 — the very word it's supposed to predict.

The solution

Before the softmax, set every score for a later position to −∞, so its attention weight becomes exactly 0.

The consequence

A decoder can be trained on every position of a sequence in parallel while behaving exactly as if it generated left to right — the basis of GPT-style training.

You should understand first

  1. Vectors
  2. Dot Product
  3. Embeddings
  4. Attention
  5. Probability and Distributions
  6. Softmax
  7. Self-Attention
  8. Causal Masking

Why it's needed

A language model learns by predicting each next token. Given "the cat sat on", position 4 (on) must predict the. If attention let on look at the, the task would be trivial and the model would learn nothing useful.

How it works

Compute all the attention scores as usual, then overwrite every score where the key comes after the query with −∞-\infty:

scoresij={qi⋅kj/dkj≤i−∞j>i\text{scores}_{ij} = \begin{cases} q_i \cdot k_j / \sqrt{d_k} & j \le i \\ -\infty & j > i \end{cases}

Softmax of −∞-\infty is exactly 0, so future tokens contribute nothing. The weight matrix becomes lower-triangular: row ii has non-zero entries only in columns 1…i1 \dots i.

The payoff: parallel training

Because of the mask, one forward pass over a 1,000-token sequence yields 1,000 valid next-token predictions — each made as if the text beyond it didn't exist. That efficiency is a large part of why decoder-only Transformers train so well at scale.

What to remember

  • Mask = set future scores to −∞ before softmax → weight 0.
  • The attention matrix becomes lower-triangular.
  • It lets one forward pass train n next-token predictions at once.

Key papers

Essential

Attention Is All You Need

Ashish Vaswani, Noam Shazeer et al. · 2017 · NeurIPS 2017

Introduced the Transformer — the architecture behind BERT, GPT and nearly every modern large language model, and later adapted to vision, audio and more.

How to read it: Section 3 is the architecture — read it with Figure 1 open. Sections 3.2.1–3.2.2 contain the attention equation. You can skim the training details on a first pass.

~1 h 15 min readarXiv:1706.03762✓ verified 2026-09-26

Watch