Skip to content
Road to Intelligence

Concept · Chapter 3: Machine Learning

Principal Component Analysis (PCA)

Should knowUnderstand15 minDifficulty

PCA finds the few directions along which data varies the most, so high-dimensional data can be summarized, compressed or plotted with little loss.

The problem

Data often has many correlated features; it's hard to visualize, slow to process and noisy.

The solution

Find orthogonal directions of maximum variance — the eigenvectors of the data's covariance matrix — and project the data onto the top few.

The consequence

A standard tool for visualization, compression and denoising — and the go-to way to plot embeddings (Chapter 6) in two dimensions.

You should understand first

  1. Vectors
  2. Dot Product
  3. Matrix Multiplication
  4. Probability and Distributions
  5. Expected Value and Variance
  6. Principal Component Analysis (PCA)

Intuition

A cloud of points shaped like a flattened cigar in 3-D is really "mostly a line". PCA finds that line (the first principal component), then the direction of next-most spread perpendicular to it, and so on. Keep the first two and you have a faithful 2-D picture of 3-D — or of 768-D — data.

How it connects to Chapter 2

The principal components are the eigenvectors of the covariance matrix; each one's eigenvalue is the variance along it. Projecting onto the top kk is a matrix multiplication by a d×kd \times k matrix.

What to remember

  • Principal components = directions of greatest variance, mutually perpendicular.
  • Computed from eigenvectors of the covariance matrix (or via SVD).
  • Keep the top k components to reduce dimensions with minimal information loss.
  • Linear: it can't unfold curved structure.

Key papers

Watch