Dot Product

Intuition

You have two vectors and want one number that says how much they agree. The dot product gives exactly that: it rewards dimensions where both vectors are large and have the same sign, and penalizes dimensions where they disagree.

Think of rating films on two axes, action and romance. You like action a lot and romance a little: you = [2, 1]. A film that's pure action is film = [3, 0]. How well does the film match you? Multiply your appetite for each quality by how much the film has of it, and add up.

Tiny numeric example

1. 1

   Multiply matching entries

   $[1, 2] \cdot [3, 4]$ → first entries: $1 \times 3 = 3$; second entries: $2 \times 4 = 8$.

2. 2

   Add them up

   $3 + 8 = 11$. That's the dot product: one number, not a vector.

3. 3

   Try an opposite pair

   $[1, 2] \cdot [-1, -2] = -1 - 4 = -5$. Pointing opposite ways gives a negative score.

4. 4

   Try a perpendicular pair

   $[1, 2] \cdot [2, -1] = 2 - 2 = 0$. At right angles, the vectors share nothing: zero.

The equation

$$\mathbf{a} \cdot \mathbf{b} \;=\; \sum_{i=1}^{d} a_i\, b_i \;=\; \lVert \mathbf{a} \rVert\, \lVert \mathbf{b} \rVert \cos\theta$$

That scaling matters. A very long vector can score high against everything. When you only care about direction, divide the lengths out — that's cosine similarity:

$$\cos\theta = \frac{\mathbf{a}\cdot\mathbf{b}}{\lVert\mathbf{a}\rVert\,\lVert\mathbf{b}\rVert}$$

Try it

Where it appears in AI

• A neuron computes $\mathbf{w}\cdot\mathbf{x} + b$: how much the input $\mathbf{x}$ matches the pattern stored in its weights $\mathbf{w}$.
• Matrix multiplication is a grid of dot products — every row of one matrix against every column of the other. A neural-network layer is thousands of dot products at once.
• Embedding search / RAG: the documents whose vectors have the highest dot product (or cosine) with your query vector are the ones retrieved.
• Attention: each token's query is dot-producted with every other token's key. High score = "that token is relevant to me". This is the $QK^\top$ in the attention equation.