Scaling laws, in the context of machine learning, are empirical relationships describing how a model's performance improves in a predictable, roughly power-law fashion as three quantities increase: the number of parameters in the model, the amount of training data it sees, and the total compute spent on training. Rather than a single law, the term refers to a body of research establishing that loss on held-out data tends to decrease smoothly and predictably as these quantities scale up together, which allows researchers to forecast the performance of a much larger model before training it.
Kaplan et al., 2020
The foundational paper, "Scaling Laws for Neural Language Models," was published by Jared Kaplan and collaborators at OpenAI in January 2020. It found that language model loss followed power-law relationships with model size, dataset size, and compute, largely independent of specific architectural details such as network depth or width, and that larger models trained on the same amount of data were consistently more sample-efficient. A widely drawn implication at the time was that, for a fixed compute budget, it was better to train very large models on comparatively modest amounts of data rather than training smaller models to convergence, a conclusion that shaped the design of models such as GPT-3.
Chinchilla, 2022
A 2022 paper by Hoffmann and colleagues at Google DeepMind, introducing a model called Chinchilla, revised this picture. By training a large number of models of different sizes on different amounts of data and comparing them at equal compute budgets, the authors found that the original scaling laws underestimated how much training data a model needs relative to its parameter count; many contemporary large models were, in this framing, undertrained relative to their size. The Chinchilla paper's compute-optimal recipe, which recommended scaling model size and training data roughly in proportion, became an influential reference point for how labs allocated compute in subsequent Pretraining runs.
Consequences and debate
Scaling laws helped justify the massive capital investment behind the growth of foundation models, since a predictable relationship between compute and performance gives investors and labs a basis for forecasting returns on ever-larger training runs. They also raised the question of whether scaling could continue indefinitely, given finite high-quality training data and the physical and financial limits described informally by extensions of Moore's law to AI compute growth. Concerns about running out of easily available high-quality text data, sometimes called a data wall, pushed research toward Synthetic data and toward new scaling axes beyond pretraining, including Test-time compute, the practice of spending more compute at inference to improve output quality on a given problem rather than only scaling the model itself. The relationship between raw scaling and qualitatively new capabilities remains debated, and is closely tied to research on Emergent abilities, which asks whether some capabilities appear abruptly at particular scales rather than improving smoothly along with loss.