Chinchilla is a family of large language models (LLMs) developed by the research team at Google DeepMind, presented in March 2022. The family was designed to investigate the scaling laws of language models, building on earlier work with the Gopher model family. Chinchilla models are transformer-based and were trained to achieve better performance under fixed compute budgets, leading to a widely cited recommendation for training large models efficiently.
The name "Chinchilla" reflects its role as a further development over the previous Gopher model family. Both families were trained to explore how model size and training data volume interact. Chinchilla claimed to outperform GPT-3 while using significantly fewer parameters, simplifying downstream use by requiring less computational power for inference and fine-tuning.
Scaling Laws and Training
The central finding of the Chinchilla research was that, for a given compute budget, model size and training tokens should scale proportionally. Specifically, if the model size is doubled, the number of training tokens should also be doubled. This principle was derived from analyzing the training of previously employed language models. DeepMind used this hypothesis to train Chinchilla, which has 70 billion parameters and was trained on 1.4 trillion tokens, four times more data than Gopher's 300 billion tokens, at a similar overall compute cost.
This compute-optimal approach contrasts with earlier practices that often prioritized increasing model parameters alone. The Chinchilla team recommended that using larger, higher-quality training datasets can lead to better results on downstream tasks, even when model size is not increased. The findings have influenced subsequent training methodologies across the field of AI research.
Performance and Benchmarks
Chinchilla achieved an average accuracy of 67.5% on the Measuring Massive Multitask Language Understanding (MMLU) benchmark, which is 7% higher than Gopher's performance. This improvement was notable given that Gopher had 280 billion parameters, four times more than Chinchilla. The model's efficiency made it practical for a wider range of applications, as it required less compute for both inference and fine-tuning.
As of January 12, 2023, Chinchilla was still in the testing phase, indicating that its full capabilities and limitations were being evaluated. The model's success on benchmarks like MMLU demonstrated that compute-optimal training could yield competitive or superior results compared to larger models trained with less data.
Architecture
Both the Gopher and Chinchilla families are families of transformer models. They are essentially the same as GPT-2, with different sizes and minor modifications. The Gopher family uses RMSNorm instead of LayerNorm and relative positional encoding rather than absolute positional encoding. The Chinchilla family is identical to the Gopher family but is trained with the AdamW optimizer instead of the Adam optimizer.
The Gopher family contains six models of increasing size, from 44 million parameters to 280 billion parameters, with the largest referred to as "Gopher" by default. Similar naming conventions apply to the Chinchilla family, which includes models of various sizes, with the 70-billion-parameter version being the most prominent. Table 1 of the original paper details the entire Gopher family, while Table 4 compares the 70-billion-parameter Chinchilla with Gopher 280B.
Applications and Influence
Chinchilla has been used as a foundation for other models, including the Flamingo vision-language model, which integrates visual and textual understanding. The scaling laws established by Chinchilla have become a standard reference in the field of machine learning, guiding decisions about model size and dataset size in subsequent LLM developments.
The research contributed to developing an effective training paradigm for large autoregressive language models with limited compute resources. By demonstrating that data volume is as important as model size, Chinchilla shifted the focus toward dataset quality and curation. This has had a lasting impact on how organizations like OpenAI and Anthropic approach model training, though the specific details of their internal practices are not always public.
See Also
- Gopher
- Scaling laws
- Deep learning