Concept · Chapter 6: Language Before Transformers
Neural Machine Translation
Neural machine translation trains an encoder and decoder to map a source-language sequence to a target-language sequence.
The problem
Languages differ in word order and length, so translation requires a flexible mapping between two sequences.
The solution
Learn the mapping end to end with a sequence encoder, an autoregressive decoder, and eventually attention over source positions.
The consequence
Translation became a proving ground for seq2seq, attention and then the Transformer.
You should understand first
- Text as Data
- Probability and Distributions
- Conditional Probability and Bayes' Theorem
- Probability of Sequences
- Language Modeling
- Vectors
- Dot Product
- The Turing Test
- Symbolic AI
- Logic and Rules
- Expert Systems
- Knowledge Representation
- The Knowledge-Acquisition Bottleneck
- From Rules to Learning
- Supervised, Unsupervised and Self-Supervised Learning
- Features, Labels and Tasks
- Loss Functions
- Derivatives and Gradients
- Gradient Descent
- Linear Regression
- Entropy
- Softmax
- Cross-Entropy Loss
- Logistic Regression
- The Perceptron
- Activation Functions
- The Artificial Neuron
- Embeddings
- Recurrent Neural Networks
- The Chain Rule
- Matrix Multiplication
- Multilayer Perceptron (MLP)
- The Forward Pass
- Computational Graphs and Autodiff
- Backpropagation
- Vanishing and Exploding Gradients
- LSTMs and GRUs
- Sequence-to-Sequence Models
- Attention
- Neural Machine Translation
Why translation forced better sequence models
Translation cannot be solved by assigning one class label to a sentence. "The blue house" and "la maison bleue" differ in both language and word order. A decoder has to generate a variable-length answer and decide which source detail matters for each output word.
Early recurrent encoder-decoders showed the pattern. Attention made it stronger by letting the decoder build a fresh context from the source states at each step. The original 2017 Transformer kept the encoder-decoder translation task but replaced recurrence with attention-based blocks.
This history matters even if you never build a translator: the same source-to-output pattern later appears in summarization and many other conditional generation tasks.
What to remember
- The target sentence can change word order and length.
- The decoder predicts each target unit conditioned on previous target units and the encoded source.
Key papers
Sequence to Sequence Learning with Neural Networks
Ilya Sutskever, Oriol Vinyals, Quoc V. Le · 2014 · NeurIPS 2014
Established the encoder–decoder pattern: read an input sequence into a vector, then generate an output sequence from it. Its central weakness motivated attention.
How to read it: Notice the trick of reversing the source sentence — a hint that long-range dependencies were the real problem.
Neural Machine Translation by Jointly Learning to Align and Translate
Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio · 2014 · ICLR 2015
Introduced attention in neural networks for language: instead of squeezing a sentence into one vector, the decoder looks back at every input word and decides which ones matter right now.
How to read it: Figure 3's alignment heat-maps are the best picture of 'attention' ever drawn — look at them first.