Grokking is a phenomenon observed in machine learning, particularly in deep learning, where a neural network appears to memorize training data for an extended period before abruptly transitioning to a state of genuine generalization. The term, coined by researchers at OpenAI in 2022, describes a sudden and often dramatic improvement in performance on unseen data after a prolonged phase of apparent overfitting. This delayed generalization contrasts with typical training dynamics, where models improve gradually or plateau early, and has significant implications for understanding how neural networks learn.
The phenomenon was first systematically documented in a 2022 paper by Alethea Power and colleagues at OpenAI, who studied small transformer models trained on algorithmic tasks such as modular arithmetic. They found that after the model achieved near-perfect accuracy on training data, it continued to train for many more steps with no visible improvement on validation data, only to suddenly achieve near-perfect generalization. This "grokking" event, named after Robert A. Heinlein's 1961 novel Stranger in a Strange Land (where "grok" means to understand profoundly), has since been replicated across various architectures and tasks, sparking research into the underlying mechanisms.
Characteristics and Conditions
Grokking typically occurs in models trained with specific conditions: small datasets, simple algorithmic tasks, and certain regularization techniques. The phenomenon is most pronounced when the model is trained with weight decay, which penalizes large weights, or with other forms of regularization like dropout or data augmentation. In many experiments, grokking appears only after the model has overfit the training data completely, and the transition is often sharp, occurring within a few hundred or thousand training steps after a long plateau.
The delay before grokking can be substantial, sometimes requiring orders of magnitude more training steps than initial fitting. For example, in the original modular arithmetic experiments, models took tens of thousands of steps to memorize the data, then plateaued for hundreds of thousands of steps before suddenly generalizing. This behavior is sensitive to hyperparameters: larger models tend to grok faster, while smaller models may never grok, and the presence of weight decay is often crucial.
Theoretical Explanations
Several theories have been proposed to explain grokking, though no consensus exists as of 2025. One prominent hypothesis is that grokking arises from a competition between memorization and generalization in the model's internal representations. During the plateau phase, the network encodes training examples in a complex, overfitted manner, but gradient descent gradually simplifies these representations, eventually discovering a more general solution. This process is analogous to phase transitions in physics, where a system suddenly reorganizes into a lower-energy state.
Another line of work, including studies by researchers at the University of Toronto and elsewhere, suggests that grokking is related to the geometry of the loss landscape. The model may initially be trapped in a sharp minimum that generalizes poorly, but as training continues, it escapes to a flatter minimum that generalizes better. Weight decay encourages this escape by penalizing sharp solutions. Additionally, some research has linked grokking to the emergence of modular or structured representations, such as Fourier features in arithmetic tasks, which allow the model to compute answers using general rules rather than memorized examples.
Implications for Machine Learning
Grokking challenges conventional wisdom about overfitting and generalization. In standard practice, training is stopped when validation performance plateaus, but grokking shows that continuing to train can yield dramatic improvements, even after apparent convergence. This has practical implications for training large models, where computational budgets are often limited. If grokking occurs in real-world tasks, it could mean that some models are undertrained and would benefit from longer training runs, but detecting the onset of grokking is difficult because it is unpredictable and can require excessive compute.
The phenomenon also provides a testbed for studying how neural networks transition from memorization to generalization, which is relevant to understanding the capabilities of large language models and other AI systems. Researchers have drawn parallels between grokking and the sudden emergence of abilities in large models, though the relationship remains speculative. Understanding grokking could inform techniques for accelerating learning, such as curriculum learning or adaptive regularization, and might help design models that generalize more reliably.
Related Phenomena and Research Directions
Grokking is related to other observed training dynamics, such as "double descent," where test performance first improves, then worsens, then improves again as model size grows. Both phenomena highlight the non-monotonic nature of learning in neural networks. However, grokking is distinct in that it occurs over time for a fixed model, rather than over model size. Another related concept is "critical learning periods" in biological systems, where certain experiences must occur at specific times for proper development, which some researchers have compared to the timing of grokking events.
Current research directions include investigating grokking in larger models and more complex tasks, such as natural language processing, where evidence is mixed. Some studies have reported grokking-like behavior in transformer-based language models, but others have not observed it, possibly due to differences in optimization and data scale. Researchers are also exploring how initialization, learning rate schedules, and architecture choices affect grokking, with the goal of developing predictive theories. As of 2025, grokking remains an active area of study, with open questions about its universality, mechanisms, and practical relevance.
See Also
- Deep learning
- Neural network
- Transformer (architecture)
- OpenAI
- Weight Initialization
- Curriculum Learning
- Data Augmentation
- Model Pruning
References
Power, A., Burda, Y., Edwards, H., Babuschkin, I., & Misra, V. (2022). Grokking: Generalization beyond overfitting on small algorithmic datasets. arXiv preprint arXiv:2201.02177.
Nanda, N., Chan, L., Lieberum, T., Smith, J., & Steinhardt, J. (2023). Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217.
Varma, V., Das, R., Schmidt, D., & Shah, N. (2023). Explaining grokking through circuit efficiency. arXiv preprint arXiv:2309.02390.