Skip to content
Road to Intelligence

Concept · Chapter 3: Machine Learning

Logistic Regression

Must knowImplement30 minDifficulty

Logistic regression is a linear classifier: it computes a weighted sum of the features and squashes it through a sigmoid to get a probability, trained by minimizing cross-entropy.

The problem

Linear regression predicts unbounded numbers, but a yes/no question needs a probability between 0 and 1 — and a loss suited to being right or wrong.

The solution

Pass the linear score z = w·x + b through the sigmoid σ(z) = 1/(1+e^(−z)) to get a probability, and fit the weights by gradient descent on binary cross-entropy.

The consequence

It is exactly a single artificial neuron with a sigmoid output — the bridge from classical ML to neural networks — and its multi-class form (softmax regression) is the output layer of every classifier and LLM.

You should understand first

  1. The Turing Test
  2. Symbolic AI
  3. Logic and Rules
  4. Expert Systems
  5. Knowledge Representation
  6. The Knowledge-Acquisition Bottleneck
  7. From Rules to Learning
  8. Supervised, Unsupervised and Self-Supervised Learning
  9. Vectors
  10. Features, Labels and Tasks
  11. Loss Functions
  12. Derivatives and Gradients
  13. Gradient Descent
  14. Linear Regression
  15. Probability and Distributions
  16. Entropy
  17. Softmax
  18. Cross-Entropy Loss
  19. Logistic Regression

Intuition

Compute a score by weighting the evidence — like linear regression — then turn the score into a probability. Large positive score → close to 1; large negative → close to 0; zero → 50/50. The sigmoid is the smooth S-shaped curve that does this.

Tiny numeric example

  1. Score

    Weights w = [2, −1], b = −0.5, input x = [1, 0.5]: z = 2·1 − 1·0.5 − 0.5 = 1.

  2. Probability

    σ(1) = 1 / (1 + e⁻¹) ≈ 0.73.

  3. Loss if the true label is 1

    −ln 0.73 ≈ 0.31. If the true label were 0: −ln 0.27 ≈ 1.31.

  4. Gradient

    ∂L/∂w = (p − y)·x. For y = 1: (0.73 − 1)·[1, 0.5] = [−0.27, −0.135] — so gradient descent increases both weights a little.

The equations

p=σ(w⊤x+b)=11+e−(w⊤x+b)L=−1N∑i[yiln⁡pi+(1−yi)ln⁡(1−pi)]p = \sigma(\mathbf{w}^\top\mathbf{x} + b) = \frac{1}{1 + e^{-(\mathbf{w}^\top\mathbf{x} + b)}} \qquad \mathcal{L} = -\frac{1}{N}\sum_i \big[y_i \ln p_i + (1-y_i)\ln(1-p_i)\big]

Try it

Try it · toy model

Logistic Regression Lab

Train a linear classifier with gradient descent and watch its decision boundary settle — then see it fail on XOR.

Implement8 min

Its limit — and the road to neural networks

The decision boundary is always a straight line. On XOR-shaped data no line works, and logistic regression is stuck at chance. Two ways out: hand-craft a new feature (e.g. x1⋅x2x_1 \cdot x_2) — the classical approach — or learn the features with a hidden layer — the neural-network approach of Chapter 4.

Why should I care?

As a researcher

The last layer of nearly every classifier — including the LM head choosing the next token — is multinomial logistic regression on learned features. Linear probes used in interpretability are logistic regressions too.

As an engineer

A fast, calibrated, interpretable baseline for any binary prediction: churn, fraud, click-through. If a big model can't beat it, something is wrong.

Modern systems that depend on it

  • The sigmoid neuron
  • Softmax output layers and LM heads
  • Linear probes
  • Calibration methods

Historical context

Before

Linear discriminant analysis and linear regression applied (poorly) to 0/1 outcomes; the logistic function itself dates to the 19th century.

After

Neural networks — stacks of such units — which learn the features the logistic layer consumes.

Used today

Credit scoring, medicine, ad click prediction, and as the final layer of deep classifiers and language models.

What to remember

  • p = σ(w·x + b), with σ(z) = 1 / (1 + e^(−z)).
  • The decision boundary w·x + b = 0 is a straight line (a hyperplane).
  • Trained by minimizing binary cross-entropy with gradient descent.
  • Gradient per example is (p − y)·x — elegantly simple.
  • Can't separate data like XOR without better features or a hidden layer.

Watch