Tensors and Shapes

Must knowKnow well15 minDifficulty

In deep learning a tensor is just an n-dimensional array of numbers, and keeping track of its shape — batch × sequence × features — is most of the practical work of reading and writing models.

The problem

Real models process many examples, many tokens and many features at once; vectors and matrices alone don't describe that bookkeeping.

The solution

Use arrays with more axes, give each axis a meaning, and let operations like matrix multiplication act on the last axes while the leading ones are carried along.

The consequence

Code and papers can describe whole-batch computations compactly — and shape mismatches become the most common bug you'll see.

You should understand first

Vectors
Tensors and Shapes

The idea

A scalar is one number, a vector a 1-D array, a matrix a 2-D array. A tensor, in ML usage, is any n-dimensional array. (Physicists mean something more specific; you can ignore that.)

The shapes you'll see constantly:

Data	Shape	Meaning of each axis
A batch of tabular rows	[B, F]	examples, features
A batch of images	[B, C, H, W]	examples, colour channels, height, width
LLM activations	[B, T, D]	examples, tokens, model dimension
Attention weights	[B, H, T, T]	examples, heads, query token, key token

Batched matrix multiplication

When you multiply a [B, T, D] tensor by a [D, D'] weight matrix, the matmul happens on the last axis and the result is [B, T, D'] — the same weights applied to every token of every example. Frameworks (NumPy, PyTorch) do this automatically, along with broadcasting: a [D] bias vector is stretched to add onto every row.

What to remember

Scalar → vector → matrix → tensor: 0, 1, 2, n axes.
A typical LLM activation has shape [batch, sequence, d_model].
Matmul applies to the last two axes; leading axes are batched.
Broadcasting stretches size-1 axes to match.