Concept · Chapter 6: Language Before Transformers
Sequence-to-Sequence Models
A sequence-to-sequence model uses an encoder to read one sequence and a decoder to produce another, possibly of a different length.
The problem
Translation, summarization and similar tasks need an output sequence whose length need not match the input.
The solution
An encoder turns the source sequence into a representation, and a decoder generates the target one step at a time.
The consequence
The encoder-decoder pattern generalizes beyond next-word prediction, but early versions forced all source information through one fixed-size vector.
You should understand first
- Text as Data
- Probability and Distributions
- Conditional Probability and Bayes' Theorem
- Probability of Sequences
- Language Modeling
- Vectors
- Dot Product
- The Turing Test
- Symbolic AI
- Logic and Rules
- Expert Systems
- Knowledge Representation
- The Knowledge-Acquisition Bottleneck
- From Rules to Learning
- Supervised, Unsupervised and Self-Supervised Learning
- Features, Labels and Tasks
- Loss Functions
- Derivatives and Gradients
- Gradient Descent
- Linear Regression
- Entropy
- Softmax
- Cross-Entropy Loss
- Logistic Regression
- The Perceptron
- Activation Functions
- The Artificial Neuron
- Embeddings
- Recurrent Neural Networks
- The Chain Rule
- Matrix Multiplication
- Multilayer Perceptron (MLP)
- The Forward Pass
- Computational Graphs and Autodiff
- Backpropagation
- Vanishing and Exploding Gradients
- LSTMs and GRUs
- Sequence-to-Sequence Models
Read first, then write
For translation, an encoder reads "the blue house" and a decoder writes "la maison bleue". The source and target have different word order and may have different lengths. The decoder uses its own previous output and the encoder's representation to decide what comes next.
Cho and colleagues' 2014 RNN encoder-decoder and Sutskever and colleagues' sequence-to-sequence LSTMs established influential versions of this pattern. The latter reversed source sentences during training, which shortened some input-output dependency paths. Both early approaches relied on a single fixed-size source representation passed to the decoder.
That architecture is more flexible than one classifier with a fixed-size output, but it creates the next problem: where does the detail of a long input go?
What to remember
- The encoder reads the source; the decoder writes the target. Their lengths can differ.
- Early RNN encoder-decoders passed a single final vector between the two halves.
Key papers
Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation
Kyunghyun Cho, Bart van Merrienboer et al. · 2014 · EMNLP 2014
Proposed a jointly trained RNN encoder and decoder for mapping one sequence to another, with the gated recurrent unit in the architecture.
Sequence to Sequence Learning with Neural Networks
Ilya Sutskever, Oriol Vinyals, Quoc V. Le · 2014 · NeurIPS 2014
Established the encoder–decoder pattern: read an input sequence into a vector, then generate an output sequence from it. Its central weakness motivated attention.
How to read it: Notice the trick of reversing the source sentence — a hint that long-range dependencies were the real problem.