Skip to content
Road to Intelligence

Concept · Chapter 3: Machine Learning

Regularization

Must knowKnow well20 minDifficulty

Regularization is anything that discourages a model from fitting the training data too closely — most commonly a penalty on large weights — so that it generalizes better.

The problem

Flexible models overfit: they bend to fit noise in the training data.

The solution

Add a penalty on model complexity to the loss (L2 'ridge' / weight decay, L1 'lasso'), or constrain training in other ways (early stopping, dropout, data augmentation).

The consequence

Complexity becomes a dial (λ) rather than a fixed choice; weight decay is used when training most large neural networks, and L1 yields sparse, interpretable models.

You should understand first

  1. The Turing Test
  2. Symbolic AI
  3. Logic and Rules
  4. Expert Systems
  5. Knowledge Representation
  6. The Knowledge-Acquisition Bottleneck
  7. From Rules to Learning
  8. Supervised, Unsupervised and Self-Supervised Learning
  9. Vectors
  10. Features, Labels and Tasks
  11. Loss Functions
  12. Derivatives and Gradients
  13. Gradient Descent
  14. Linear Regression
  15. Probability and Distributions
  16. Expected Value and Variance
  17. Sampling and Uncertainty
  18. Generalization, Overfitting and Underfitting
  19. Regularization

The idea

Lreg(w)=1N∑iℓ(fw(xi),yi)⏟fit the data+λ∥w∥22⏟keep weights small\mathcal{L}_{\text{reg}}(\mathbf{w}) = \underbrace{\frac{1}{N}\sum_i \ell(f_\mathbf{w}(x_i), y_i)}_{\text{fit the data}} + \underbrace{\lambda \lVert \mathbf{w} \rVert_2^2}_{\text{keep weights small}}

A wiggly degree-11 polynomial needs huge, finely balanced coefficients to pass through every point. Charge a price for large coefficients and the optimizer prefers a smoother curve that fits the data almost as well. In the Overfitting Lab, turning on even a small λ at degree 11 brings the test error back down to near the best simple model.

The family

  • L2 / ridge / weight decay — penalize squared weights. In neural networks this is usually applied as weight decay; AdamW applies it correctly for adaptive optimizers.
  • L1 / lasso — penalize absolute weights; produces sparse models where many features get weight exactly zero (built-in feature selection).
  • Early stopping — stop training when validation error starts rising.
  • Dropout (Chapter 4), data augmentation — make memorization harder.

What to remember

  • Regularized loss = data loss + λ · complexity penalty.
  • L2 (ridge / weight decay): penalize Σw²; shrinks all weights smoothly.
  • L1 (lasso): penalize Σ|w|; drives many weights to exactly zero.
  • λ is chosen on validation data.
  • Early stopping, dropout and data augmentation are regularizers too.

Key papers

Important

Decoupled Weight Decay Regularization

Ilya Loshchilov, Frank Hutter · 2017 · ICLR 2019

Introduced AdamW, the variant of Adam used to train most modern Transformers and LLMs.

~40 min readarXiv:1711.05101✓ verified 2026-09-26
Optional

Regression Shrinkage and Selection Via the Lasso

Robert Tibshirani · 1996 · Journal of the Royal Statistical Society, Series B

Introduced the lasso (L1 regularization), which shrinks weights and sets many exactly to zero — regularization and feature selection at once.

~50 min readdoi:10.1111/j.2517-6161.1996.tb02080.x✓ verified 2026-09-26

Watch