Skip to content
Road to Intelligence

Concept · Chapter 4: Neural Networks

Computational Graphs and Autodiff

Must knowKnow well15 minDifficulty

A computational graph records every elementary operation of a calculation, so that software can compute exact derivatives of the output with respect to every input automatically.

The problem

Deriving gradients by hand for every new model is slow and error-prone, and numerical approximations are too inaccurate and expensive.

The solution

Break the computation into simple operations with known local derivatives, record them as a graph during the forward pass, then walk the graph backward applying the chain rule.

The consequence

Frameworks like PyTorch and JAX compute gradients of arbitrary programs automatically — which is why researchers can try new architectures without deriving a single derivative.

You should understand first

  1. Derivatives and Gradients
  2. The Chain Rule
  3. Vectors
  4. Dot Product
  5. The Turing Test
  6. Symbolic AI
  7. Logic and Rules
  8. Expert Systems
  9. Knowledge Representation
  10. The Knowledge-Acquisition Bottleneck
  11. From Rules to Learning
  12. Supervised, Unsupervised and Self-Supervised Learning
  13. Features, Labels and Tasks
  14. Loss Functions
  15. Gradient Descent
  16. Linear Regression
  17. Probability and Distributions
  18. Entropy
  19. Softmax
  20. Cross-Entropy Loss
  21. Logistic Regression
  22. The Perceptron
  23. Activation Functions
  24. The Artificial Neuron
  25. Matrix Multiplication
  26. Multilayer Perceptron (MLP)
  27. The Forward Pass
  28. Computational Graphs and Autodiff

Tiny example

Compute L=(w⋅x+b)2L = (w \cdot x + b)^2 with w=2,x=3,b=1w = 2, x = 3, b = 1:

  1. Forward, one operation at a time

    u = w·x = 6; v = u + b = 7; L = v² = 49.

  2. Local derivatives

    ∂L/∂v = 2v = 14; ∂v/∂u = 1; ∂v/∂b = 1; ∂u/∂w = x = 3.

  3. Backward sweep (chain rule)

    ∂L/∂b = 14·1 = 14; ∂L/∂w = 14·1·3 = 42.

That's all an autodiff engine does, for millions of operations. Reverse mode is efficient because one backward sweep costs about as much as the forward pass, no matter how many parameters there are Established — the key to training huge networks.

What to remember

  • Graph nodes = simple operations (add, multiply, tanh…); edges = values.
  • Each operation knows its own local derivative.
  • Reverse mode: one backward sweep gives gradients for all parameters.
  • loss.backward() in PyTorch = reverse-mode autodiff on the recorded graph.

Watch