The Bitter Lesson and Scaling Laws

Why Raw Compute and Power-Law Curves Repeatedly Defeat Human Ingenuity

In January 2020, physicist Jared Kaplan and a team of researchers plotted the cross-entropy test loss of dozens of neural language models on a doubly logarithmic graph. The horizontal axis measured total training compute in floating-point operations (FLOPs), spanning six orders of magnitude. The vertical axis measured how accurately each model predicted the next word in unseen text. Instead of an erratic scatter of points reflecting clever architectural tweaks, the measurements fell along a ruler-straight line. Just as gas molecules obey thermodynamic pressure laws despite chaotic individual collisions, the aggregate prediction error of neural networks obeyed a smooth mathematical power law.

Ten months before Kaplan's plot appeared, reinforcement-learning pioneer Richard Sutton had diagnosed the historical pattern behind that straight line. In a short March 2019 essay titled The Bitter Lesson, Sutton reviewed seven decades of artificial intelligence research across chess, Go, speech recognition, and computer vision. In field after field, researchers spent years coding clever human domain rules: chess grandmaster heuristics, phonetic dictionaries, or hand-crafted visual edge detectors. Every time, those human-engineered features were demolished as soon as hardware compute grew cheap enough to run general statistical learning and search. Sutton concluded that the clearest lesson from seventy years of artificial intelligence research is that general methods leveraging massive computation ultimately prevail by a wide margin.

Why does human domain engineering lose over time? Handcrafted priors bake in simplified approximations of how humans introspect about their own thinking, which creates a rigid ceiling once data and compute expand. General learning algorithms discover statistical structures across thousands of dimensions that human minds cannot verbalize.

In March 2022, Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, and their colleagues at DeepMind refined Kaplan's curves. They trained over four hundred language models ranging from 70 million to over 16 billion parameters across datasets of 5 billion to 500 billion tokens. Their Chinchilla scaling law proved that test loss $L$ as a function of model parameter count $N$ and dataset token count $D$ follows a two-term additive power law above the irreducible entropy $E$ of natural text:

Because the empirical exponents $\alpha \approx 0.34$ and $\beta \approx 0.28$ are nearly equal, minimizing loss under a fixed budget of floating-point operations ($C \approx 6ND$) requires scaling model parameters and training tokens in roughly equal one-to-one proportion. Prior laboratories had built giant parameter counts while starving them of training text. To test their equation, DeepMind trained a 70-billion-parameter model named Chinchilla on 1.4 trillion tokens. That dataset was four times larger than the text fed to their earlier 280-billion-parameter Gopher model, while using the exact same compute budget. Chinchilla outperformed the four-times-larger Gopher model across virtually every reasoning, reading, and coding benchmark.

This continuous power-law drop also explains the apparent mystery of emergent capabilities. Suppose a multi-step arithmetic or logic puzzle requires five sequential reasoning steps to be right simultaneously. If single-step accuracy rises smoothly from 50 percent to 90 percent as overall loss falls along the power law, five-step accuracy jumps from 3 percent ($0.5^5$) to 59 percent ($0.9^5$). What looks like a sudden phase transition on a pass-fail test is driven by smooth exponential compression underneath. When engineers plot a language model's next-word prediction error against training compute on a logarithmic chart, the points fall along a straight line spanning seven orders of magnitude before any sudden task capability appears.