Concept · Chapter 6: Language Before Transformers
One-Hot Encoding
A one-hot vector represents a vocabulary item with a 1 in its own position and 0 everywhere else.
The problem
A token ID is an arbitrary category label, but a neural network needs a numeric input.
The solution
Give each vocabulary item its own vector coordinate and turn on exactly that coordinate.
The consequence
This represents identity without inventing an ordering, but every two different words are equally far apart and the vectors grow with the vocabulary.
You should understand first
- Text as Data
- Vectors
- One-Hot Encoding
The useful limitation
With vocabulary [cat, dog, coffee], the vectors are cat = [1,0,0], dog = [0,1,0], and coffee = [0,0,1]. One-hot encoding avoids the false suggestion that ID 3 is greater than ID 2. But and . It cannot express that the first pair is more similar.
The representation is also sparse. For a vocabulary of 50,000 words, each input vector has 50,000 entries even though only one is nonzero. In practice a model does not need to allocate that full vector: multiplying a one-hot vector by an embedding matrix is equivalent to selecting one row from the matrix.
What to remember
- For V words, every one-hot vector has V coordinates and exactly one 1.
- Different one-hot words have dot product 0, whether they are related or not.