Concept · Chapter 4: Neural Networks
The Perceptron
The perceptron (1958) is a single artificial neuron that outputs 1 if a weighted sum of its inputs exceeds a threshold, with a simple rule for learning the weights from mistakes.
The problem
Could a machine learn to recognize patterns from examples, instead of having its rules or connections designed by hand?
The solution
Compute w·x + b; output 1 if positive, else 0. After each mistake, nudge the weights toward the correct answer: w ← w + η(y − ŷ)x.
The consequence
It learns any linearly separable pattern — but, as Minsky and Papert showed in 1969, no single perceptron can learn XOR, a limitation that helped stall neural-network research for over a decade.
You should understand first
- Vectors
- Dot Product
- The Turing Test
- Symbolic AI
- Logic and Rules
- Expert Systems
- Knowledge Representation
- The Knowledge-Acquisition Bottleneck
- From Rules to Learning
- Supervised, Unsupervised and Self-Supervised Learning
- Features, Labels and Tasks
- Loss Functions
- Derivatives and Gradients
- Gradient Descent
- Linear Regression
- Probability and Distributions
- Entropy
- Softmax
- Cross-Entropy Loss
- Logistic Regression
- The Perceptron
The idea
Frank Rosenblatt's perceptron took the McCulloch–Pitts neuron (1943) and added learning. It computes a weighted sum of its inputs and fires if the sum crosses a threshold. When it gets an example wrong, it adjusts its weights a little toward the right answer. If the two classes can be separated by a straight line (a hyperplane), this procedure is guaranteed to find one Established.
The limit
XOR — output 1 when exactly one input is on — can't be separated by any single line. A perceptron can't learn it, however long it trains. Minsky and Papert's 1969 book analysed such limits rigorously. The fix — layers of neurons with a way to train the hidden ones — existed in outline but wasn't widely known and used until backpropagation was popularized in 1986.
In the Neural Network Lab, set hidden units to 0 on XOR: a single neuron stalls near chance. Add a hidden layer and it solves it.
What to remember
- Output = step(w·x + b): a hard yes/no linear classifier.
- Learning rule: on each mistake, w ← w + η (y − ŷ) x.
- Guaranteed to converge if the data is linearly separable.
- Cannot represent XOR — that needs a hidden layer.
Key papers
A logical calculus of the ideas immanent in nervous activity
Warren S. McCulloch, Walter Pitts · 1943 · The Bulletin of Mathematical Biophysics
The first mathematical model of a neuron as a logic unit — the seed of both neural networks and the idea that thought could be computation.
How to read it: Historically important but hard to read today. The idea — neurons as threshold logic gates — is what matters.
The perceptron: A probabilistic model for information storage and organization in the brain.
F. Rosenblatt · 1958 · Psychological Review
Introduced the perceptron — a neuron model that learns its weights from examples. Every neural network descends from it.