Cerebras-GPT is a family of open-source large language models developed by Cerebras Systems, a company known for its wafer-scale neural network accelerators. The models were designed to achieve compute-optimal performance, meaning they were trained to balance model size and training data volume according to the Chinchilla scaling law, which states that for a given compute budget, larger models with proportionally more data yield better performance. Cerebras-GPT was released in March 2023, with the goal of demonstrating the efficiency of Cerebras's CS-2 system for training state-of-the-art models at scale.
The family includes models with parameter counts ranging from 111 million to 13 billion, specifically: 111M, 256M, 590M, 1.3B, 2.7B, 6.7B, and 13B. All models were trained on the same public dataset, the Pile, which is a diverse collection of text from books, academic papers, and web content. The training used a fixed compute budget per model, following the Chinchilla optimal ratio of roughly 20 tokens per parameter, ensuring that each model was trained on the appropriate amount of data to maximize performance for its size. The models were trained using the Adam optimizer with a cosine learning rate schedule and gradient clipping to stabilize training.
Architecture and Training
Cerebras-GPT models use the standard Transformer architecture, specifically the decoder-only variant, which is common for generative language models. Each model employs multi-head attention, layer normalization, and learned positional embeddings. The models were trained on the CS-2 system, which integrates a single silicon wafer with over 850,000 cores, enabling high-throughput training without the need for distributed training across multiple GPUs. This architecture allowed Cerebras to train the models in a fraction of the time and energy compared to traditional GPU clusters, as the wafer-scale design reduces communication overhead.
Training was conducted in bfloat16 precision, and the models were trained from scratch without any pre-existing checkpoints. The training process used a batch size of 256 sequences, each with a length of 2048 tokens, and the total training tokens for each model were determined by the Chinchilla formula. For example, the 13B model was trained on approximately 260 billion tokens, while the 111M model was trained on about 2.2 billion tokens. The training data was shuffled and processed to ensure diversity, and the models were evaluated on standard benchmarks such as language modeling perplexity and downstream tasks.
Performance and Benchmarks
Cerebras-GPT models were evaluated on several benchmarks, including LAMBADA, HellaSwag, and Winogrande, which test language understanding and reasoning. The results showed that the models performed competitively with other open-source models of similar size, such as those from OpenAI and Anthropic, despite being trained with a simpler recipe. For instance, the 13B model achieved a LAMBADA accuracy of 72.3%, which is comparable to other 13B models available at the time. The models also demonstrated strong performance on the Pile's held-out validation set, with perplexity decreasing as model size increased, as expected.
One notable aspect of Cerebras-GPT is its focus on compute efficiency. The company reported that training the 13B model on the CS-2 took approximately 10 days, whereas a comparable training run on a cluster of 512 NVIDIA A100 GPUs would take around 30 days. This efficiency is attributed to the wafer-scale design, which eliminates the need for model parallelism and reduces inter-chip communication. The models were released under the Apache 2.0 license, allowing unrestricted use and modification, which was a deliberate move to promote research and development in the open-source community.
Comparison with Other Models
At the time of release, Cerebras-GPT stood out among open-source models for its adherence to the Chinchilla scaling law. Many earlier models, such as GPT-3, were trained with more data than compute-optimal, leading to inefficiencies. Cerebras-GPT was one of the first families to explicitly follow the compute-optimal approach, which meant that each model was trained on exactly the right amount of data for its parameter count. This approach was validated by the fact that the models achieved performance comparable to or better than models trained with suboptimal data-to-parameter ratios.
Compared to later models like LLaMA (which was released later in 2023), Cerebras-GPT was smaller in maximum size but offered a more systematic study of scaling. The models were also notable for their reproducibility, as the training code and configurations were open-sourced, allowing researchers to replicate the results. However, the models were not instruction-tuned or fine-tuned for chat, so they were primarily intended for research on scaling laws and efficient training rather than deployment in applications.
Impact and Legacy
Cerebras-GPT contributed to the broader trend of democratizing large language models by providing a family of models that could be run on modest hardware. The smallest model (111M) could be fine-tuned on a single GPU, while the largest (13B) required a multi-GPU setup but was still accessible to many research labs. The release also highlighted the potential of alternative hardware architectures, such as wafer-scale integration, in reducing the cost and energy footprint of training large models.
The models were used in subsequent research on curriculum learning and data augmentation techniques, as their open-source nature allowed for easy experimentation. Cerebras Systems continued to develop larger models, such as the Cerebras-GPT 7B and later the Cerebras-GPT 13B variants, but the original family remains a reference point for compute-optimal training. The company also used the experience to inform the development of its next-generation hardware, the CS-3, which was announced in 2024.
Availability and Usage
Cerebras-GPT models are available on Hugging Face and through the Cerebras Model Zoo, which provides pre-trained checkpoints and inference scripts. The models can be used with standard machine learning frameworks like PyTorch, and the training code is available on GitHub. The Apache 2.0 license permits commercial use, making the models attractive for startups and enterprises looking to deploy language models without licensing fees. However, users should be aware that the models were not aligned with human preferences, so they may produce biased or harmful outputs, and appropriate safeguards are recommended for deployment.
In summary, Cerebras-GPT is a significant contribution to the field of artificial intelligence, demonstrating that compute-optimal training is achievable with specialized hardware and providing a valuable resource for researchers studying scaling laws and model efficiency.