Concept · Chapter 6: Language Before Transformers
The Fixed-Vector Bottleneck
Early encoder-decoder models compressed every detail of the source sequence into one fixed-size vector before decoding.
The problem
A decoder translating a long sentence may need a precise source word that one compressed summary has lost.
The solution
Keep the encoder's per-position states and let the decoder retrieve the relevant ones at each output step.
The consequence
This bottleneck motivated neural attention, which made source information accessible without a single final-state choke point.
You should understand first
- Text as Data
- Probability and Distributions
- Conditional Probability and Bayes' Theorem
- Probability of Sequences
- Language Modeling
- Vectors
- Dot Product
- The Turing Test
- Symbolic AI
- Logic and Rules
- Expert Systems
- Knowledge Representation
- The Knowledge-Acquisition Bottleneck
- From Rules to Learning
- Supervised, Unsupervised and Self-Supervised Learning
- Features, Labels and Tasks
- Loss Functions
- Derivatives and Gradients
- Gradient Descent
- Linear Regression
- Entropy
- Softmax
- Cross-Entropy Loss
- Logistic Regression
- The Perceptron
- Activation Functions
- The Artificial Neuron
- Embeddings
- Recurrent Neural Networks
- The Chain Rule
- Matrix Multiplication
- Multilayer Perceptron (MLP)
- The Forward Pass
- Computational Graphs and Autodiff
- Backpropagation
- Vanishing and Exploding Gradients
- LSTMs and GRUs
- Sequence-to-Sequence Models
- The Fixed-Vector Bottleneck
One suitcase for every trip
An early translation encoder could read forty words, then hand the decoder one vector. The decoder had to reconstruct each translated word using that same compressed source summary. Increasing the source length did not increase the size of the suitcase. Information useful for a late output word could be hard to preserve through the whole encoding path.
This is an architectural bottleneck, not a theorem that a finite vector cannot encode a long sentence. In practice, finite model capacity, optimization and the long information path made it difficult. Bahdanau and colleagues reported that a fixed-vector encoder-decoder degraded on long sentences and proposed a learned alignment mechanism instead.
Keep a state for every input position. When the decoder needs the next word, score those states and combine the relevant ones. That is attention.
Why should I care?
As a researcher
The bottleneck explains the experimental problem Bahdanau attention was built to address.
As an engineer
It is a reminder to inspect an architecture's information path, not only its parameter count.
Modern systems that depend on it
- attention
- neural translation
- Transformers
Historical context
Before
The decoder received one vector summarizing the entire source, regardless of source length.
After
Attention let each decoder step draw a different weighted summary from all encoder positions.
Used today
The general question remains: which parts of a long input can a model access, and through what path?
What to remember
- A fixed-size state must summarize a source sequence whose length can grow.
- The issue is loss of accessible detail, not that a vector mathematically cannot encode any long sequence.
- Attention keeps the per-position states and chooses a fresh weighted context for each output step.
Key papers
Sequence to Sequence Learning with Neural Networks
Ilya Sutskever, Oriol Vinyals, Quoc V. Le · 2014 · NeurIPS 2014
Established the encoder–decoder pattern: read an input sequence into a vector, then generate an output sequence from it. Its central weakness motivated attention.
How to read it: Notice the trick of reversing the source sentence — a hint that long-range dependencies were the real problem.
Neural Machine Translation by Jointly Learning to Align and Translate
Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio · 2014 · ICLR 2015
Introduced attention in neural networks for language: instead of squeezing a sentence into one vector, the decoder looks back at every input word and decides which ones matter right now.
How to read it: Figure 3's alignment heat-maps are the best picture of 'attention' ever drawn — look at them first.