OPT (Open Pre-trained Transformer) is a family of decoder-only large language models developed by Meta AI and released in May 2022. The suite spans parameter sizes from 125 million to 175 billion, with the largest model, OPT-175B, matching the scale of GPT-3. Unlike many contemporaneous models, OPT was made publicly available with its weights, training code, and detailed logs, aiming to democratize access to large-scale language model research.
The architecture of OPT follows the standard Transformer (architecture) decoder design, incorporating improvements such as layer normalization and residual connections. The models were trained on a diverse corpus of 180 billion tokens, including sources like Common Crawl, Wikipedia, and BooksCorpus. The training process emphasized computational efficiency and stability, using techniques like dynamic loss scaling and gradient clipping.
Development and Release
OPT was introduced in a paper titled "OPT: Open Pre-trained Transformer Language Models" by Susan Zhang and colleagues at Meta AI. The release included not only the model weights but also the training code and a detailed log of the training process, which was unprecedented for models of this scale. The goal was to encourage reproducibility and collaboration within the artificial intelligence research community.
The models were made available under a non-commercial license, with access granted through a request process. The release was accompanied by a demo and documentation. The largest model, OPT-175B, required significant computational resources, but smaller variants (e.g., 125M, 350M, 1.3B, 2.7B, 6.7B, 13B, 30B, 66B) were also provided to accommodate different research needs.
Architecture and Training
OPT uses a decoder-only architecture with causal attention, similar to GPT-3. Key design choices include pre-normalization (applying layer norm before each sub-layer), ReLU activation instead of GELU, and learned positional embeddings. The training data was filtered and deduplicated, and the models were trained using Adam optimization with a cosine learning rate schedule.
The training of OPT-175B took approximately 992 GPU-hours on NVIDIA A100 GPUs, which was notably more efficient than comparable models due to optimization techniques. The training logs documented challenges such as loss spikes and the mitigation strategies employed.
Performance and Evaluation
OPT models were evaluated on a range of natural language processing benchmarks, including SuperGLUE, SQuAD, and OpenBookQA. While OPT-175B performed competitively with GPT-3 on many tasks, it showed some differences, such as slightly lower performance on code generation tasks. The models also exhibited few-shot learning capabilities, improving with the number of examples provided.
In zero-shot and few-shot settings, OPT-175B achieved strong results on question answering and commonsense reasoning tasks. However, like other large models, it could produce biased or harmful outputs, and the paper discussed limitations and ethical considerations.
Impact and Legacy
OPT was a landmark release in the open-source AI community, setting a precedent for transparency in large-scale model development. It influenced subsequent open models such as BLOOM and LLaMA. The availability of OPT enabled researchers to study model behavior, Fine-tuning, and Mechanistic interpretability without the barriers of proprietary models.
The release also sparked discussions about the risks of open-sourcing powerful models, including potential misuse. Meta AI implemented a responsible release strategy, including a use case policy and access controls. Despite these concerns, OPT remains a widely used baseline in academic research and a reference point for open model development.
See Also
References
- Zhang, S., et al. (2022). "OPT: Open Pre-trained Transformer Language Models." arXiv:2205.01068.
- Meta AI. (2022). "Introducing OPT."