Concept · Chapter 4: Neural Networks
Dropout
Dropout randomly switches off a fraction of neurons at each training step, so the network can't rely on any single unit and must learn redundant, more general features.
The problem
Large networks can memorize their training data, with neurons co-adapting in fragile ways.
The solution
During training, zero each unit's output with probability p (and rescale the rest); at test time, use every unit.
The consequence
A simple, widely used regularizer — in effect training an ensemble of thinned networks that share weights.
You should understand first
- The Turing Test
- Symbolic AI
- Logic and Rules
- Expert Systems
- Knowledge Representation
- The Knowledge-Acquisition Bottleneck
- From Rules to Learning
- Supervised, Unsupervised and Self-Supervised Learning
- Vectors
- Features, Labels and Tasks
- Loss Functions
- Derivatives and Gradients
- Gradient Descent
- Linear Regression
- Probability and Distributions
- Expected Value and Variance
- Sampling and Uncertainty
- Generalization, Overfitting and Underfitting
- Regularization
- Dot Product
- Entropy
- Softmax
- Cross-Entropy Loss
- Logistic Regression
- The Perceptron
- Activation Functions
- The Artificial Neuron
- Matrix Multiplication
- Multilayer Perceptron (MLP)
- Dropout
How it works
At each training step, every hidden unit is dropped with probability — its output set to zero — and the surviving outputs are scaled by so their expected sum is unchanged. Each step trains a different random sub-network; at test time the full network approximates their average.
Large language models are often pretrained with little or no dropout, since they see most training text only about once and overfitting is less of a concern Interpretation; dropout remains common in fine-tuning and in smaller models trained for many epochs.
What to remember
- Training: randomly zero units with probability p (e.g. 0.1–0.5).
- Inference: use all units (outputs appropriately scaled).
- Prevents co-adaptation; acts like averaging many sub-networks.
- The original Transformer used p = 0.1.
Key papers
Dropout: A Simple Way to Prevent Neural Networks from Overfitting
Nitish Srivastava, Geoffrey Hinton et al. · 2014 · Journal of Machine Learning Research
Dropout — randomly switching off units during training — became a standard, simple regularizer for neural networks.