A curated set, not a bibliography. Each paper is listed with the problem it solved and what it changed, and every title, author list and link was checked against arXiv. Read the essentials first, in chronological order — they tell the story on their own.
Replaced the unanswerable question 'Can machines think?' with a testable one — the imitation game — and anticipated learning machines decades early.
Problem
'Thinking' has no agreed definition, so the question of machine intelligence seemed impossible to discuss scientifically.
What was new
An operational test based on conversation, a systematic rebuttal of objections to machine intelligence, and a proposal to build 'child machines' that learn rather than being fully programmed.
Yann LeCun, Léon Bottou et al. · 1998 · Proceedings of the IEEE
The LeNet paper: convolutional networks trained end-to-end with gradient descent for handwriting recognition, deployed commercially for reading cheques.
Problem
Handwriting recognition relied on hand-designed feature extractors plus a trainable classifier.
What was new
Learn the features too: convolutional networks trained end to end, plus whole systems trained with gradients.
A short, practical essay on the lessons ML practitioners learn the hard way: generalization is what counts, data beats cleverness, and intuition fails in high dimensions.
Problem
The folk knowledge that separates successful ML projects from failed ones was rarely written down.
What was new
Twelve compact lessons — overfitting, the curse of dimensionality, feature engineering, more data vs smarter algorithms, and more.
How to read it: The best single reading for Chapter 3. Read it after the chapter; much of it will click.
~30 min readdoi:10.1145/2347736.2347755✓ verified 2026-09-26
Tomas Mikolov, Kai Chen et al. · 2013 · ICLR 2013 (workshop)
Showed that simple, fast models trained on billions of words produce word vectors whose geometry captures meaning — the idea behind every embedding you use today.
Problem
Neural language models learned good word representations but were too slow to train on very large corpora.
What was new
Two stripped-down architectures (CBOW and skip-gram) that drop the expensive hidden layer, making it practical to learn embeddings from huge datasets.
Ilya Sutskever, Oriol Vinyals, Quoc V. Le · 2014 · NeurIPS 2014
Established the encoder–decoder pattern: read an input sequence into a vector, then generate an output sequence from it. Its central weakness motivated attention.
Problem
Standard neural networks need fixed-size inputs and outputs, but translation maps sequences to sequences of different lengths.
What was new
A deep LSTM encoder compresses the source sentence into one vector; a second LSTM decodes the translation from it, trained end to end.
Introduced attention in neural networks for language: instead of squeezing a sentence into one vector, the decoder looks back at every input word and decides which ones matter right now.
Problem
Encoder–decoder models squeezed the whole source sentence into a single fixed-length vector, and translation quality fell sharply on long sentences.
What was new
A learned alignment: at each output step the model scores every encoder state, normalizes the scores with softmax, and uses the weighted average as context.
Introduced the Transformer — the architecture behind BERT, GPT and nearly every modern large language model, and later adapted to vision, audio and more.
Problem
Recurrent models process tokens one after another, which limits parallel training and forces distant words to interact through many sequential steps.
What was new
Dropped recurrence entirely: stacks of multi-head self-attention and feed-forward layers, with positional encodings, residual connections and layer normalization.
How to read it: Section 3 is the architecture — read it with Figure 1 open. Sections 3.2.1–3.2.2 contain the attention equation. You can skim the training details on a first pass.
~1 h 15 min readarXiv:1706.03762✓ verified 2026-09-26
Tom B. Brown, Benjamin Mann et al. · 2020 · NeurIPS 2020
GPT-3 (175B parameters) showed that a large enough language model can perform new tasks from a few examples in its prompt, without any gradient updates.
Problem
Each new NLP task needed its own labelled dataset and fine-tuning run.
What was new
Scale a decoder-only Transformer by ~100× and evaluate zero-, one- and few-shot 'in-context learning' across many tasks.
InstructGPT: the supervised fine-tuning + reward model + RL recipe that turned GPT-3 into an instruction-following assistant, and the template for ChatGPT.
Problem
Pretrained language models continue text; they don't reliably follow instructions or behave helpfully.
What was new
Fine-tune on human demonstrations, train a reward model on human rankings, then optimize the model against it with PPO.
C. E. Shannon · 1948 · Bell System Technical Journal
Founded information theory: it defined entropy as a measure of uncertainty and showed how much any message can be compressed. Cross-entropy loss and perplexity come straight from here.
Problem
There was no precise way to measure information, or to say how efficiently it could be encoded and transmitted over a noisy channel.
What was new
Entropy H = −Σ p log p as the average information of a source, plus limits on compression and reliable communication. It even includes early statistical models of English text built from n-gram frequencies.
How to read it: Don't read it cover to cover. Part I (sections 1–7) contains entropy and the famous 'series of approximations to English' — a 1948 language model.
~2 h readdoi:10.1002/j.1538-7305.1948.tb01338.x✓ verified 2026-09-26
John McCarthy, Marvin L. Minsky et al. · 1955 · Reprinted in AI Magazine 27(4), 2006
The document that named the field 'artificial intelligence' and set out its founding ambition.
Problem
Scattered work on thinking machines had no shared name, agenda or community.
What was new
A two-month study proposed on the conjecture that every aspect of learning or intelligence can in principle be described precisely enough for a machine to simulate it.
Kurt Hornik, Maxwell Stinchcombe, Halbert White · 1989 · Neural Networks
Proved that a network with a single hidden layer can approximate essentially any continuous function, given enough units — the 'universal approximation' result.
Problem
Were multi-layer networks fundamentally limited, as single-layer perceptrons had been shown to be?
What was new
A proof that standard feedforward networks with one hidden layer are universal approximators.
Random forests — many decision trees trained on random subsets of data and features, then averaged — remain one of the strongest methods for tabular data.
Problem
Single decision trees overfit easily and are unstable: small data changes produce very different trees.
What was new
Average many decorrelated trees, each grown on a bootstrap sample with a random subset of features at each split.
~50 min readdoi:10.1023/A:1010933404324✓ verified 2026-09-26
Yoshua Bengio, Réjean Ducharme et al. · 2003 · Journal of Machine Learning Research
Learned word representations and next-word probabilities jointly, so similar words could help the model generalize to word sequences it had never counted.
Problem
An n-gram table sees most possible word sequences zero times, and counting alone cannot share evidence between similar words.
What was new
A neural probability model that looks up a distributed vector for each context word and learns those vectors with the prediction task.
Shachar Kaufman, Saharon Rosset et al. · 2012 · ACM Transactions on Knowledge Discovery from Data
Named and systematized data leakage — information in training data that won't exist at prediction time — one of the most common ways ML results turn out to be fake.
Problem
Models that looked excellent in evaluation failed in deployment because their training data contained hints about the answer.
What was new
A formal definition of leakage, a catalogue of real examples, and methods to detect and avoid it.
~45 min readdoi:10.1145/2382577.2382579✓ verified 2026-09-26
Colin Raffel, Noam Shazeer et al. · 2019 · JMLR 2020
Framed every NLP task as text in, text out, using an encoder–decoder Transformer — and ran a huge, careful set of ablations that is still a model of empirical method.
Problem
Transfer-learning results were hard to compare because every paper changed many things at once.
What was new
One text-to-text format for all tasks, plus a systematic study of objectives, architectures and data.
An openly released reasoning model, with a detailed account of training long chains of reasoning mainly through reinforcement learning on verifiable problems.
Problem
How reasoning models were trained was largely undisclosed.
What was new
Large-scale RL with rule-based rewards on math and code, plus distillation of the resulting reasoning into smaller open models.