Concept · Chapter 3: Machine Learning
Hand-Crafted Features vs Learned Features
Classical ML learns only the final mapping from features to labels — people design the features — while deep learning learns the features too, directly from raw data.
The problem
Classical models are only as good as their input features, and designing good features for images, audio or text took years of expert effort per domain.
The solution
First, experts hand-crafted features (edge detectors, word counts, spectral coefficients). Then neural networks learned layered features automatically from raw pixels, waveforms and tokens.
The consequence
Learning the representation removed the main bottleneck of classical ML — and is the central reason deep learning overtook it on perception and language (Chapter 4).
You should understand first
- The Turing Test
- Symbolic AI
- Logic and Rules
- Expert Systems
- Knowledge Representation
- The Knowledge-Acquisition Bottleneck
- From Rules to Learning
- Supervised, Unsupervised and Self-Supervised Learning
- Vectors
- Features, Labels and Tasks
- Loss Functions
- Derivatives and Gradients
- Gradient Descent
- Linear Regression
- Probability and Distributions
- Entropy
- Softmax
- Cross-Entropy Loss
- Logistic Regression
- Hand-Crafted Features vs Learned Features
The same bottleneck, one level down
Chapter 1 ended with a problem: hand-written rules don't scale. Classical machine learning fixed it — for the final step. Given good features, a logistic regression learns the weights. But someone still had to design the features:
| Domain | Hand-crafted features (classical era) |
|---|---|
| Images | edge and gradient histograms (e.g. SIFT, HOG) |
| Speech | spectral coefficients (e.g. MFCCs) |
| Text | word counts, TF-IDF, n-grams |
| Tabular | ratios, aggregates, time-since-last-event… |
Domingos (2012) observed that feature engineering is often the most important factor in whether an ML project succeeds Interpretation — and it is expert work, slow and domain-specific: the knowledge-acquisition bottleneck again, one level down.
Try the classical fix
In the Logistic Regression Lab, XOR-shaped data defeats a linear model. The classical fix is to invent a feature — makes XOR linearly separable. That works when you know what to invent.
The deep-learning fix
Instead of inventing features, learn them: put layers of adjustable feature detectors in front of the classifier and train everything together. The first layers learn simple patterns, later layers combine them into more abstract ones. That's the promise of neural networks — Chapter 4.
Why should I care?
As a researcher
'Representation learning' is the idea that unifies deep learning, embeddings and foundation models; this is where it starts.
As an engineer
On tabular business data, good hand-built features still matter enormously. On images, audio and text, you'll almost always start from a pretrained model's learned features (embeddings) instead.
Modern systems that depend on it
- Representation learning
- Embeddings
- Pretrained models and transfer learning
Historical context
Before
Expert-designed features: SIFT and HOG descriptors for images, MFCCs for audio, bag-of-words and TF-IDF for text.
After
Deep networks that learn features end to end; later, pretrained models whose learned features (embeddings) are reused across tasks.
Used today
Hand-crafted features dominate tabular ML pipelines; learned features dominate vision, speech and language.
What to remember
- Classical ML: humans design features, the model learns the last step.
- Deep learning: the model learns features from raw data too.
- Hand-crafted features were a knowledge-engineering bottleneck in disguise.
- Learned features (embeddings) can be reused across tasks.
Key papers
A few useful things to know about machine learning
Pedro Domingos · 2012 · Communications of the ACM
A short, practical essay on the lessons ML practitioners learn the hard way: generalization is what counts, data beats cleverness, and intuition fails in high dimensions.
How to read it: The best single reading for Chapter 3. Read it after the chapter; much of it will click.
Gradient-based learning applied to document recognition
Yann LeCun, Léon Bottou et al. · 1998 · Proceedings of the IEEE
The LeNet paper: convolutional networks trained end-to-end with gradient descent for handwriting recognition, deployed commercially for reading cheques.
How to read it: Long (46 pages). Sections I–II explain why learned features beat hand-designed ones — the heart of Chapter 4.