Skip to content
Road to Intelligence

Concept · Chapter 4: Neural Networks

Weight Initialization

Should knowUnderstand10 minDifficulty

Initialization sets the random starting weights at a scale that keeps signals and gradients roughly the same size from layer to layer, so deep networks can start learning at all.

The problem

Weights that start too small make signals shrink layer after layer; too large and they explode or saturate the activations.

The solution

Draw initial weights with variance scaled to the layer's width: about 1/n (Xavier/Glorot, for tanh-like units) or 2/n (He/Kaiming, for ReLU).

The consequence

A one-line change that made much deeper networks trainable; frameworks now apply sensible defaults automatically.

You should understand first

  1. Derivatives and Gradients
  2. The Chain Rule
  3. Vectors
  4. Dot Product
  5. The Turing Test
  6. Symbolic AI
  7. Logic and Rules
  8. Expert Systems
  9. Knowledge Representation
  10. The Knowledge-Acquisition Bottleneck
  11. From Rules to Learning
  12. Supervised, Unsupervised and Self-Supervised Learning
  13. Features, Labels and Tasks
  14. Loss Functions
  15. Gradient Descent
  16. Linear Regression
  17. Probability and Distributions
  18. Entropy
  19. Softmax
  20. Cross-Entropy Loss
  21. Logistic Regression
  22. The Perceptron
  23. Activation Functions
  24. The Artificial Neuron
  25. Matrix Multiplication
  26. Multilayer Perceptron (MLP)
  27. The Forward Pass
  28. Computational Graphs and Autodiff
  29. Backpropagation
  30. Vanishing and Exploding Gradients
  31. Weight Initialization

Why a random start, and why the scale matters

If every weight started equal, every neuron in a layer would compute the same thing and receive the same gradient — they'd never differentiate. So weights start random. But a sum of nn random terms has variance proportional to nn: with weights of variance σ2\sigma^2, a layer's outputs have variance about nσ2n\sigma^2 times its inputs'. Choosing σ2≈1/n\sigma^2 \approx 1/n keeps it steady; ReLU zeroes half its inputs, so it needs 2/n2/n.

Wij∼N ⁣(0,1nin)    (Xavier-style)Wij∼N ⁣(0,2nin)    (He, for ReLU)W_{ij} \sim \mathcal{N}\!\left(0, \tfrac{1}{n_{\text{in}}}\right) \;\;\text{(Xavier-style)} \qquad\qquad W_{ij} \sim \mathcal{N}\!\left(0, \tfrac{2}{n_{\text{in}}}\right) \;\;\text{(He, for ReLU)}

Compare the three initialization settings in the vanishing gradient lab with ReLU selected.

What to remember

  • Random init breaks symmetry (identical weights would learn identical features).
  • Scale matters: keep activation and gradient variance constant across layers.
  • Xavier/Glorot: Var(w) ≈ 1/n. He/Kaiming (ReLU): Var(w) ≈ 2/n.

Key papers