Skip to content
Road to Intelligence

Concept · Chapter 3: Machine Learning

Linear Regression

Must knowImplement30 minDifficulty

Linear regression predicts a number as a weighted sum of the features plus a constant, choosing the weights that minimize the average squared error on the training data.

The problem

Given examples with numeric outcomes, we want a simple, interpretable rule that predicts the outcome for new examples.

The solution

Assume the prediction is w·x + b, measure error with mean squared error, and find the w and b that minimize it — exactly (a formula) or iteratively (gradient descent).

The consequence

It is the template for all of machine learning — model, loss, optimization — and the building block of every neural-network layer, which is a linear map followed by a nonlinearity.

You should understand first

  1. The Turing Test
  2. Symbolic AI
  3. Logic and Rules
  4. Expert Systems
  5. Knowledge Representation
  6. The Knowledge-Acquisition Bottleneck
  7. From Rules to Learning
  8. Supervised, Unsupervised and Self-Supervised Learning
  9. Vectors
  10. Features, Labels and Tasks
  11. Loss Functions
  12. Derivatives and Gradients
  13. Gradient Descent
  14. Linear Regression

Intuition

Plot house size against price and draw the line that passes "closest" to the points. That line is a model: for a new house, read its price off the line. Linear regression is the precise version: closest means smallest average squared vertical distance.

Tiny numeric example

  1. Data

    Three houses: (size 1, price 2), (2, 3), (3, 5) — in hundreds of m² and hundreds of thousands.

  2. Try a line

    ŷ = 1.5·x + 0.33 predicts 1.83, 3.33, 4.83. Errors: −0.17, 0.33, −0.17.

  3. Score it

    MSE = (0.028 + 0.111 + 0.028) / 3 ≈ 0.056. This line happens to be the least-squares best; any other line scores worse.

The equations

y^=w⊤x+bL(w,b)=1N∑i=1N(y^i−yi)2w∗=(X⊤X)−1X⊤y\hat{y} = \mathbf{w}^\top \mathbf{x} + b \qquad \mathcal{L}(\mathbf{w}, b) = \frac{1}{N}\sum_{i=1}^{N} \big(\hat{y}_i - y_i\big)^2 \qquad \mathbf{w}^* = (X^\top X)^{-1} X^\top \mathbf{y}

"Linear" is about the weights, not the curve

Add features x2,x3,…x^2, x^3, \dots and the same machinery fits curves — the model is still linear in its weights. That's exactly what the lab below does, and it's where the most important idea in this chapter shows up: the more flexible the model, the better it fits the training data — and, past a point, the worse it predicts new data.

Try it · toy model

Overfitting Lab

Fit curves of increasing complexity to 12 noisy points. Training error keeps falling; error on new data falls, then soars. Then add regularization.

Know well8 min

Why should I care?

As a researcher

Linear models are the baseline every new method should beat, and the setting where most ML theory (generalization, regularization, double descent) is first worked out.

As an engineer

For forecasting and tabular prediction, a well-featured linear model is often a strong, cheap, explainable first answer — and the 'linear layer' is the most common operation in deep learning.

Modern systems that depend on it

  • Logistic regression
  • Neural-network layers (Wx + b)
  • Ridge and lasso
  • Linear probes in interpretability

Historical context

Before

Least squares dates to Legendre and Gauss in the early 1800s, used to fit astronomical observations.

After

Regularized linear models (ridge, lasso), generalized linear models such as logistic regression, and neural networks that stack linear maps with nonlinearities.

Used today

Forecasting, pricing, A/B-test analysis, scientific data analysis — and inside every neural network as the linear layer.

What to remember

  • Model: ŷ = w·x + b (a weighted sum of features).
  • Loss: mean squared error, (1/N) Σ (ŷ − y)².
  • Solve exactly with the normal equations, or iteratively with gradient descent.
  • Weights are interpretable: change in prediction per unit change of a feature (holding others fixed).
  • A curve is still 'linear regression' if the features are non-linear (x, x², x³…).

Watch