# OPT

OPT (Open Pre-trained Transformer) is a suite of decoder-only language models developed by Meta AI, released in May 2022. The models range from 125 million to 175 billion parameters and are notable for being fully open-sourced, including weights, code, and training logs.

OPT (Open Pre-trained Transformer) is a family of decoder-only [large language model](https://www.wikiprompt.org/wiki/large-language-model)s developed by [Meta AI](https://www.wikiprompt.org/wiki/meta-ai) and released in May 2022. The suite spans parameter sizes from 125 million to 175 billion, with the largest model, OPT-175B, matching the scale of [GPT-3](https://www.wikiprompt.org/wiki/gpt-3). Unlike many contemporaneous models, OPT was made publicly available with its weights, training code, and detailed logs, aiming to democratize access to large-scale language model research.

The architecture of OPT follows the standard [transformer](https://www.wikiprompt.org/wiki/transformer) decoder design, incorporating improvements such as [layer normalization](https://www.wikiprompt.org/wiki/layer-normalization) and residual connections. The models were trained on a diverse corpus of 180 billion tokens, including sources like [Common Crawl](https://www.wikiprompt.org/wiki/common-crawl), [Wikipedia](https://www.wikiprompt.org/wiki/wikipedia), and BooksCorpus. The training process emphasized computational efficiency and stability, using techniques like dynamic loss scaling and [gradient clipping](https://www.wikiprompt.org/wiki/gradient-clipping).

## Development and Release

OPT was introduced in a paper titled "OPT: Open Pre-trained Transformer Language Models" by Susan Zhang and colleagues at [Meta AI](https://www.wikiprompt.org/wiki/meta-ai). The release included not only the model weights but also the training code and a detailed log of the training process, which was unprecedented for models of this scale. The goal was to encourage reproducibility and collaboration within the [artificial intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence) research community.

The models were made available under a non-commercial license, with access granted through a request process. The release was accompanied by a demo and documentation. The largest model, OPT-175B, required significant computational resources, but smaller variants (e.g., 125M, 350M, 1.3B, 2.7B, 6.7B, 13B, 30B, 66B) were also provided to accommodate different research needs.

## Architecture and Training

OPT uses a decoder-only architecture with causal attention, similar to [GPT-3](https://www.wikiprompt.org/wiki/gpt-3). Key design choices include pre-normalization (applying layer norm before each sub-layer), ReLU activation instead of GELU, and learned positional embeddings. The training data was filtered and deduplicated, and the models were trained using Adam optimization with a cosine learning rate schedule.

The training of OPT-175B took approximately 992 [GPU](https://www.wikiprompt.org/wiki/gpu)-hours on NVIDIA A100 GPUs, which was notably more efficient than comparable models due to optimization techniques. The training logs documented challenges such as loss spikes and the mitigation strategies employed.

## Performance and Evaluation

OPT models were evaluated on a range of [natural language processing](https://www.wikiprompt.org/wiki/natural-language-processing) benchmarks, including [SuperGLUE](https://www.wikiprompt.org/wiki/superglue), [SQuAD](https://www.wikiprompt.org/wiki/squad), and [OpenBookQA](https://www.wikiprompt.org/wiki/openbookqa). While OPT-175B performed competitively with [GPT-3](https://www.wikiprompt.org/wiki/gpt-3) on many tasks, it showed some differences, such as slightly lower performance on code generation tasks. The models also exhibited [few-shot learning](https://www.wikiprompt.org/wiki/few-shot-learning) capabilities, improving with the number of examples provided.

In zero-shot and few-shot settings, OPT-175B achieved strong results on question answering and commonsense reasoning tasks. However, like other large models, it could produce biased or harmful outputs, and the paper discussed limitations and ethical considerations.

## Impact and Legacy

OPT was a landmark release in the open-source AI community, setting a precedent for transparency in large-scale model development. It influenced subsequent open models such as [BLOOM](https://www.wikiprompt.org/wiki/bloom) and [LLaMA](https://www.wikiprompt.org/wiki/llama). The availability of OPT enabled researchers to study model behavior, [fine-tuning](https://www.wikiprompt.org/wiki/fine-tuning), and [interpretability](https://www.wikiprompt.org/wiki/interpretability) without the barriers of proprietary models.

The release also sparked discussions about the risks of open-sourcing powerful models, including potential misuse. Meta AI implemented a responsible release strategy, including a use case policy and access controls. Despite these concerns, OPT remains a widely used baseline in academic research and a reference point for open model development.

## See Also

- [GPT-3](https://www.wikiprompt.org/wiki/gpt-3)
- [LLaMA](https://www.wikiprompt.org/wiki/llama)
- [BLOOM](https://www.wikiprompt.org/wiki/bloom)
- [Transformer](https://www.wikiprompt.org/wiki/transformer)
- [Meta AI](https://www.wikiprompt.org/wiki/meta-ai)

## References

- Zhang, S., et al. (2022). "OPT: Open Pre-trained Transformer Language Models." arXiv:2205.01068.
- Meta AI. (2022). "Introducing OPT."

---
Source: https://www.wikiprompt.org/wiki/opt
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-07T02:35:05.568977+00:00