Concept · Chapter 6: Language Before Transformers
Recurrent Neural Networks
An RNN reads a sequence one step at a time, updating a hidden state that carries information from earlier steps.
The problem
A fixed n-gram or feed-forward context window cannot use a clue that appeared far earlier in a sentence.
The solution
Reuse the same transition function at each step, combining the new input with the previous hidden state.
The consequence
The model can represent variable-length histories, but computation is sequential and training signals can decay over long paths.
You should understand first
- Text as Data
- Probability and Distributions
- Conditional Probability and Bayes' Theorem
- Probability of Sequences
- Language Modeling
- Vectors
- Dot Product
- The Turing Test
- Symbolic AI
- Logic and Rules
- Expert Systems
- Knowledge Representation
- The Knowledge-Acquisition Bottleneck
- From Rules to Learning
- Supervised, Unsupervised and Self-Supervised Learning
- Features, Labels and Tasks
- Loss Functions
- Derivatives and Gradients
- Gradient Descent
- Linear Regression
- Entropy
- Softmax
- Cross-Entropy Loss
- Logistic Regression
- The Perceptron
- Activation Functions
- The Artificial Neuron
- Embeddings
- Recurrent Neural Networks
Read, update, repeat
An RNN receives an embedding for the current word and a hidden state summarizing what came before. It computes a new state, often written
The same , and are used at each time step. That lets one network process a sequence of any length. At step 20, however, the model has only the state carried through steps 1 to 19, not direct access to their inputs.
For training, draw the repeated cell once per word and run ordinary backpropagation through that unrolled graph. This is backpropagation through time. An early word's gradient must pass through many repeated transitions, multiplying derivatives along the way. It can shrink toward zero or grow explosively, just as Chapter 4 showed for deep networks.
Why should I care?
As a researcher
Unrolling an RNN makes sequence depth and gradient flow visible; this is the architectural problem attention later changes.
As an engineer
The hidden state explains streaming sequence processing and why recurrent training cannot parallelize positions like a Transformer.
Modern systems that depend on it
- LSTMs and GRUs
- sequence-to-sequence translation
- attention
Historical context
Before
Fixed-window models had to discard all words outside a chosen recent context.
After
A learned state can summarize an arbitrary prefix, though it must be updated in order and can forget distant information.
Used today
Recurrent designs still appear in streaming and state-space sequence models, although Transformers dominate large language modeling.
What to remember
- The same weights are reused at every sequence step.
- Hidden state is a learned summary of the prefix, not a perfect transcript.
- Backpropagation through time sends training error backward across the unrolled steps.
- Long chains make gradients vanish or explode, and positions cannot be computed simultaneously.
Key papers
Learning long-term dependencies with gradient descent is difficult
Yoshua Bengio, Patrice Simard, Paolo Frasconi · 1994 · IEEE Transactions on Neural Networks
Showed why gradients vanish or explode when trained across many steps — the core obstacle for deep and recurrent networks.