2019
The Bitter Lesson
Why every clever hand-built AI eventually loses to raw compute — the lesson the field keeps relearning.
“The biggest lesson that can be read from 70 years of AI research is that general methods that leverage computation are ultimately the most effective.”
Knowledge loses to compute
Sutton argues researchers keep embedding human insight into systems (chess heuristics, linguistic features, vision pipelines) and keep getting overtaken by search and learning at scale. The bitter part: our cleverness wastes time.
History as evidence
Chess, Go, speech, vision, language: the arc repeats. Methods that use more data and compute win. Methods that encode expert structure plateau.
After transformers
Read in 2019 it sounded like reinforcement learning's manifesto. After GPT and scaling laws it reads like prophecy. Whether it stays true forever is another question Sutton would probably leave to compute.
- collection
The idea lineage of the model you talked to this morning, in reading order.
← previous · 2019
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
next · 2020 →
Language Models are Few-Shot Learners