Skip to content
Road to Intelligence

Concept · Chapter 4: Neural Networks

The Forward Pass

Must knowImplement15 minDifficulty

The forward pass is computing a network's output from its input: layer by layer, multiply by weights, add biases, apply activations.

The problem

Given a network's weights and an input, we need its prediction — and, for training, the intermediate values that backpropagation will reuse.

The solution

Feed the input through each layer in order, storing each layer's pre-activation z and activation a, and compute the loss at the end.

The consequence

It's all a network does at inference time — and during training it's half of each step, with the stored activations reused by the backward pass (the main reason training needs so much memory).

You should understand first

  1. Vectors
  2. Dot Product
  3. The Turing Test
  4. Symbolic AI
  5. Logic and Rules
  6. Expert Systems
  7. Knowledge Representation
  8. The Knowledge-Acquisition Bottleneck
  9. From Rules to Learning
  10. Supervised, Unsupervised and Self-Supervised Learning
  11. Features, Labels and Tasks
  12. Loss Functions
  13. Derivatives and Gradients
  14. Gradient Descent
  15. Linear Regression
  16. Probability and Distributions
  17. Entropy
  18. Softmax
  19. Cross-Entropy Loss
  20. Logistic Regression
  21. The Perceptron
  22. Activation Functions
  23. The Artificial Neuron
  24. Matrix Multiplication
  25. Multilayer Perceptron (MLP)
  26. The Forward Pass

The recipe

a(0)=x,z(l)=W(l)a(l−1)+b(l),a(l)=ϕ(z(l)),y^=σ(z(L))\mathbf{a}^{(0)} = \mathbf{x}, \qquad \mathbf{z}^{(l)} = W^{(l)}\mathbf{a}^{(l-1)} + \mathbf{b}^{(l)}, \qquad \mathbf{a}^{(l)} = \phi\big(\mathbf{z}^{(l)}\big), \qquad \hat{y} = \sigma\big(z^{(L)}\big)

In the lab, select a data point and choose 1. Forward pass: you'll see the input values, each hidden unit's output and the final probability — the same computation, repeated per layer, that a large model performs billions of times per prediction.

Why it costs memory in training

Backpropagation needs every layer's z\mathbf{z} and a\mathbf{a} from the forward pass. A large model processing a long sequence must keep all of them until the backward pass — which is why activation memory, not just weights, limits batch sizes (Chapter 9), and why tricks like activation checkpointing recompute some of them instead.

What to remember

  • For each layer: z = Wa_prev + b, a = φ(z).
  • The last layer produces logits → softmax/sigmoid → probabilities → loss.
  • Inference = forward pass only.
  • Training stores activations for the backward pass (memory cost).

Watch