Concept · Chapter 6: Language Before Transformers
Neural Language Model
A neural language model learns word vectors and a probability function together, so similar contexts can support one another.
The problem
A counted phrase unseen in training has no evidence, even when its words resemble words in familiar phrases.
The solution
Look up dense vectors for the context words, feed them to a neural network, and train the vectors and next-word probabilities jointly.
The consequence
Generalization can follow similarity in a learned space instead of requiring exact count-table matches.
You should understand first
Sharing evidence between similar words
Suppose training contains "the cat is sleepy" but never "the kitten is sleepy". A trigram table treats cat and kitten as unrelated IDs. A neural model learns a vector for each word while learning to predict the next one. If cat and kitten occur in similar contexts, their vectors may become close and the model can give the new phrase a sensible probability.
Bengio and colleagues' 2003 model still used a fixed number of preceding words. It did not solve long-distance memory. Its important move was replacing exact symbolic matches with learned, distributed representations that let one training example inform nearby cases.
What to remember
- Bengio et al. (2003) learned the word representation and next-word model together.
- A fixed context window remained; the breakthrough was sharing statistical strength through vectors.
Key papers
A Neural Probabilistic Language Model
Yoshua Bengio, Réjean Ducharme et al. · 2003 · Journal of Machine Learning Research
Learned word representations and next-word probabilities jointly, so similar words could help the model generalize to word sequences it had never counted.
How to read it: Read the abstract and Figure 1 first: the embedding lookup and the probability model are learned together.
Watch
Andrej Karpathy
Building makemore Part 2: MLP
Implements the Bengio et al. 2003 neural language model at character level: an embedding lookup, a hidden layer and a softmax over the next character.