Concept · Chapter 4: Neural Networks
Representation Learning
Representation learning means the network learns its own features: each layer transforms the data into a new representation, from simple patterns in early layers to abstract concepts in later ones.
The problem
Classical ML depended on experts hand-designing features for every domain — slow, expensive and limited by what people could think of.
The solution
Train a deep network end to end on raw inputs so that its hidden layers discover whatever intermediate features make the task solvable.
The consequence
Deep networks overtook hand-engineered pipelines in vision, speech and language — and the learned representations themselves (embeddings) became reusable assets, the foundation of pretrained and foundation models.
You should understand first
- The Turing Test
- Symbolic AI
- Logic and Rules
- Expert Systems
- Knowledge Representation
- The Knowledge-Acquisition Bottleneck
- From Rules to Learning
- Supervised, Unsupervised and Self-Supervised Learning
- Vectors
- Features, Labels and Tasks
- Loss Functions
- Derivatives and Gradients
- Gradient Descent
- Linear Regression
- Probability and Distributions
- Entropy
- Softmax
- Cross-Entropy Loss
- Logistic Regression
- Hand-Crafted Features vs Learned Features
- Dot Product
- The Perceptron
- Activation Functions
- The Artificial Neuron
- Matrix Multiplication
- Multilayer Perceptron (MLP)
- Representation Learning
The idea
A network's hidden layers are feature detectors that nobody designed. Training shapes them into whatever makes the final prediction easy. In the Neural Network Lab, each hidden unit on XOR ends up detecting one diagonal region — features the output layer can then combine linearly. That's representation learning at the smallest possible scale.
At scale: a hierarchy
Visualizations of convolutional image networks show early layers responding to edges and colour blobs, middle layers to textures and parts, and later layers to object-like patterns Established. This hierarchy is often described as the network building abstractions step by step, though individual units rarely correspond to cleanly human-nameable concepts Interpretation (Chapter 16's interpretability sections go further).
Why it won
No feature bottleneck
The same recipe — layers + backprop + data — works for pixels, audio and text.Scales with data and compute
More data and bigger networks kept improving the learned features; hand-crafted pipelines plateaued.Features transfer
A network trained on one large task learns representations useful for many others — the idea behind pretrained embeddings and, eventually, foundation models.
Where this leads
Why should I care?
As a researcher
Nearly every modern method is about learning better representations — self-supervised pretraining, contrastive learning, embeddings — and interpretability asks what those representations contain.
As an engineer
Using a pretrained model's embeddings instead of building features by hand is now the default for unstructured data; it's why a few lines of code can give you a strong text or image classifier.
Modern systems that depend on it
- Embeddings
- Transfer learning and fine-tuning
- Foundation models
- Interpretability of features
Historical context
Before
Hand-crafted features (edge detectors, MFCCs, bag-of-words) with shallow learned classifiers on top.
After
Deep CNNs for vision (2012 onward), learned word embeddings (2013), and pretrained Transformers whose representations transfer to many tasks.
Used today
Every modern AI system — image, speech and text models, and all LLMs — relies on learned representations.
What to remember
- Each layer re-represents its input; later layers build on earlier ones.
- Features are learned from data, not designed by hand.
- In image networks: edges → textures → parts → objects (roughly).
- Learned representations (embeddings) transfer to new tasks.
- This is the central idea behind 'deep' learning.
Key papers
ImageNet Classification with Deep Convolutional Neural Networks
Alex Krizhevsky, Ilya Sutskever, Geoffrey E. Hinton · 2012 · NeurIPS 2012
AlexNet won ImageNet 2012 by a wide margin and triggered the deep-learning era: big data plus GPUs plus deep networks.
Gradient-based learning applied to document recognition
Yann LeCun, Léon Bottou et al. · 1998 · Proceedings of the IEEE
The LeNet paper: convolutional networks trained end-to-end with gradient descent for handwriting recognition, deployed commercially for reading cheques.
How to read it: Long (46 pages). Sections I–II explain why learned features beat hand-designed ones — the heart of Chapter 4.