Gopher is a family of large language models developed by the research team at Google DeepMind. The family consists of six transformer-based models of increasing size, ranging from 44 million parameters to 280 billion parameters, with the largest model referred to as "Gopher" by default. Gopher was presented in late 2021 and trained to investigate the scaling laws of large language models, forming the basis for the subsequent Chinchilla model family introduced in March 2022.
The Gopher family was designed as part of a broader research effort into how model performance scales with parameters and training data. The models are essentially the same architecture as GPT-2, with minor modifications, including the use of RMSNorm instead of LayerNorm and relative positional encoding rather than absolute positional encoding. The largest Gopher model, with 280 billion parameters, was trained on approximately 300 billion tokens.
Scaling Laws and Training
The development of Gopher was motivated by the question of how to allocate compute resources optimally when training large language models. Prior to Gopher, many models, including OpenAI's GPT-3, were trained with a focus on increasing model size while keeping training data relatively fixed. The Gopher research team, however, found through empirical studies that for a given compute budget, the optimal model size and number of training tokens follow a specific relationship: if the model size is doubled, the number of training tokens should also be doubled. This finding, later formalized in the Chinchilla paper, suggested that many existing models were undertrained.
Gopher's training used a large corpus of text data, and the model demonstrated strong performance on a range of natural language processing benchmarks. However, its performance on the Measuring Massive Multitask Language Understanding (MMLU) benchmark was 60.5% average accuracy, a figure that would later be surpassed by its successor Chinchilla.
Architecture and Variants
The Gopher family includes six models with parameter counts of 44 million, 117 million, 417 million, 1.4 billion, 7.1 billion, and 280 billion. All models share the same core architecture based on the transformer architecture, specifically similar to GPT-2 but with the noted modifications. The use of RMSNorm, a variant of layer normalization that is computationally more efficient, and relative positional encoding, which can improve generalization to longer sequences, were key differences from earlier transformer models.
The largest Gopher model was trained using a distributed training setup, leveraging thousands of accelerators. The training process was resource-intensive, and the model required significant computational power for both inference and fine-tuning, a limitation that motivated the later development of Chinchilla.
Performance and Evaluation
Gopher was evaluated on a variety of benchmarks, including language modeling, reading comprehension, and question answering. It showed strong capabilities in many areas, but its performance was not uniformly superior to smaller models on all tasks. For instance, on some reasoning and knowledge-intensive tasks, Gopher performed well, but it also exhibited limitations in areas such as factual consistency and common-sense reasoning.
Compared to GPT-3, Gopher achieved competitive or better results on several benchmarks, but the differences were not always dramatic. The research team noted that Gopher's performance gains were often modest relative to the large increase in model size, which further highlighted the importance of training data scaling.
Legacy and Successor
Gopher's primary legacy is its contribution to understanding scaling laws in language models. The insights gained from training Gopher directly informed the design of Chinchilla, a model family with 70 billion parameters trained on 1.4 trillion tokens. Chinchilla, which was presented in March 2022, achieved higher average accuracy (67.5% on MMLU) than Gopher while using a similar compute budget, demonstrating that the scaling law recommendations were effective.
The Chinchilla family shares the same architecture as Gopher but was trained with the AdamW optimizer instead of Adam. Gopher also served as a foundation for other models, such as the Flamingo vision-language model, which utilized components from the Gopher family. As of January 2023, Chinchilla was still in the testing phase, while Gopher had been superseded by its more efficient successor.