Derivatives and Gradients

Must knowKnow well30 minDifficulty

A derivative measures how much a function's output changes when you nudge its input, and the gradient collects those rates for every input at once — pointing in the direction of steepest increase.

The problem

To improve a model we need to know, for each of its many parameters, whether increasing it would make the error go up or down, and by how much.

The solution

Take the derivative with respect to each parameter (a partial derivative) and collect them into the gradient vector; moving against the gradient decreases the function fastest.

The consequence

Training any differentiable model becomes a mechanical procedure: compute the gradient of the loss, step against it, repeat.

Intuition

You're standing on a hillside in fog. You can't see the valley, but you can feel the slope under your feet. The derivative is that slope: nudge your position a little and see how much your height changes.

Tiny numeric example

A function
$f(x) = x^2$
Nudge by h = 0.1
$f(3.1) = 9.61$
Nudge by h = 0.001
$f(3.001) = 9.006001$
The limit
$h \to 0$

From one input to many: the gradient

A model's loss depends on all its parameters. The partial derivative $\partial L / \partial w_i$ asks how the loss changes when only $w_i$ is nudged. Stack them all:

f'(x) = \lim_{h \to 0} \frac{f(x+h) - f(x)}{h} \qquad\qquad \nabla L(\mathbf{w}) = \left[ \frac{\partial L}{\partial w_1}, \frac{\partial L}{\partial w_2}, \dots, \frac{\partial L}{\partial w_n} \right]

Try it

Slopes and Steps

Shrink a secant line until it becomes the tangent — the derivative — then use it to take a gradient-descent step downhill.

Know well6 min

Where it appears in AI

Learning = following gradients downhill (gradient descent).
Backpropagation (Chapter 4) is an efficient way to compute the gradient of the loss with respect to millions of weights, using the chain rule.
Diagnosing training: gradients that shrink to nearly zero (vanishing) or blow up (exploding) are among the classic failure modes of deep networks.

Why should I care?

As a researcher

Nearly every learning algorithm in modern AI is gradient-based; papers about optimization, stability and training dynamics are papers about gradients.

As an engineer

Exploding or vanishing gradients, NaN losses and learning-rate choices are gradient problems. Frameworks compute gradients for you, but you debug them.

Modern systems that depend on it

Gradient descent and every optimizer
Backpropagation
Training every neural network
Saliency maps and attribution methods

Historical context

Before

Calculus was developed by Newton and Leibniz in the late 17th century; gradient methods for optimization date back to Cauchy in the 19th.

After

Automatic differentiation made gradients of huge programs cheap to compute, enabling deep learning at scale.

Used today

Every training step of every neural network computes a gradient with respect to all of its parameters — billions of partial derivatives at once in an LLM.

What to remember

Derivative f′(x) = the limit of (f(x + h) − f(x)) / h as h → 0 — the local slope.
Positive derivative: increasing x increases f. Negative: decreases it.
Partial derivative: vary one input, hold the others fixed.
Gradient ∇f = vector of all partial derivatives; it points uphill.
Step against the gradient to go downhill.

Watch

17 min

3Blue1Brown

The essence of calculus

Rebuilds the idea of a derivative from scratch, visually, without assuming you remember school calculus.

Must know