Skip to content
Road to Intelligence

Concept · Chapter 2: The Math Toolkit

The Chain Rule

Must knowKnow well25 minDifficulty

The chain rule says the rate of change through a chain of functions is the product of the rates of change of each link — which is exactly how gradients flow backward through the layers of a network.

The problem

A neural network is a long composition of functions; we need the derivative of the final loss with respect to a weight buried many steps earlier.

The solution

Multiply the local derivatives along the path: how the loss changes with the output, times how the output changes with the hidden value, times how the hidden value changes with the weight.

The consequence

Gradients of arbitrarily deep compositions can be computed from simple local pieces — the idea behind backpropagation and automatic differentiation.

You should understand first

  1. Derivatives and Gradients
  2. The Chain Rule

Intuition: gears

Three gears are linked. Gear B turns 3 times for each turn of gear A; gear C turns 2 times for each turn of B. How many times does C turn per turn of A? 3×2=63 \times 2 = 6. Rates along a chain multiply. That's the whole chain rule.

Tiny numeric example

  1. A two-step computation

    h=3wh = 3w (a "layer"), then L=h2L = h^2 (a "loss"). At w=1w = 1: h=3h = 3, L=9L = 9.

  2. Local derivatives

    dL/dh=2h=6dL/dh = 2h = 6 and dh/dw=3dh/dw = 3.

  3. Multiply along the chain

    dL/dw=6×3=18dL/dw = 6 \times 3 = 18.

  4. Check directly

    L=(3w)2=9w2L = (3w)^2 = 9w^2, so dL/dw=18w=18dL/dw = 18w = 18 at w=1w = 1. ✓

The equation

dLdw=dLdh⋅dhdw∂L∂w=∂L∂hn⋅∂hn∂hn−1⋯∂h2∂h1⋅∂h1∂w\frac{dL}{dw} = \frac{dL}{dh} \cdot \frac{dh}{dw} \qquad\qquad \frac{\partial L}{\partial w} = \frac{\partial L}{\partial h_n} \cdot \frac{\partial h_n}{\partial h_{n-1}} \cdots \frac{\partial h_2}{\partial h_1} \cdot \frac{\partial h_1}{\partial w}

Where it appears in AI

  • Backpropagation: the gradient for every weight is a product of local derivatives along the paths from that weight to the loss.
  • Vanishing gradients: multiply 50 factors of 0.5 and you get about 10−1510^{-15} — early layers barely learn. Multiply 50 factors of 1.5 and you get about 6×1086 \times 10^{8} — training explodes.
  • Residual connections add a path whose local derivative is 1, which is why they make deep networks trainable (see residual connections).

Why should I care?

As a researcher

Backpropagation is the chain rule applied systematically; vanishing and exploding gradients are what happens when you multiply many small or large factors together.

As an engineer

Understanding that gradients are products along paths explains why deep networks need careful initialization, normalization and residual connections.

Modern systems that depend on it

  • Backpropagation
  • Automatic differentiation (PyTorch, JAX)
  • Vanishing/exploding gradient analysis
  • Residual connections

Historical context

Before

The chain rule is centuries-old calculus (Leibniz).

After

Reverse-mode automatic differentiation (backpropagation) applied it efficiently to networks with millions of parameters; it became the standard way to train them from the 1980s on.

Used today

Every call to loss.backward() in PyTorch applies the chain rule through the whole computation graph.

What to remember

  • If y = f(g(x)), then dy/dx = f′(g(x)) · g′(x).
  • Rates along a chain multiply.
  • Backprop: start from the loss and multiply local derivatives backward, layer by layer.
  • Many factors < 1 → vanishing gradients; many factors > 1 → exploding gradients.

Key papers

Essential

Learning representations by back-propagating errors

David E. Rumelhart, Geoffrey E. Hinton, Ronald J. Williams · 1986 · Nature

Showed that backpropagation lets multi-layer networks learn useful internal representations — the algorithm that still trains every neural network.

How to read it: Only four pages in Nature. Read it after the chain-rule concept page.

~25 min readdoi:10.1038/323533a0✓ verified 2026-09-26

Watch