Jurassic-1 is a family of large language models (LLMs) developed by the Israeli artificial intelligence company AI21 Labs. Released in August 2021, it was one of the first commercially available LLMs to rival the scale of OpenAI's GPT-3, with the largest variant, Jurassic-1 Jumbo, containing 178 billion parameters. The models are designed for a wide range of natural language processing tasks, including text generation, summarization, question answering, and code generation, and are accessible via a commercial API.
Jurassic-1 is built on the Transformer (architecture) architecture, specifically an autoregressive decoder-only model, similar to GPT-3. It was trained on a large corpus of publicly available text and code, with a focus on high-quality filtering and deduplication. The model family includes two main variants: Jurassic-1 Jumbo (178B parameters) and Jurassic-1 Large (7.5B parameters), both of which share the same architecture but differ in scale. The models use a tokenizer with a vocabulary of 250,000 tokens, which is larger than GPT-3's 50,000-token vocabulary, allowing for more efficient encoding of text.
Architecture and Training
The Jurassic-1 models employ a standard decoder-only Transformer (architecture) with multi-head attention and feed-forward layers. They use Layer Normalization and a residual connection structure, with a configuration similar to other large-scale LLMs. The training process utilized a combination of Adam optimization, Gradient Clipping, and a learning rate schedule with warmup and cosine decay. The models were trained on a dataset of approximately 300 billion tokens, sourced from web pages, books, and other public text, with a significant portion in English but also including other languages.
AI21 Labs emphasized the use of a custom data filtering pipeline to remove low-quality or duplicated content. The training was performed on a cluster of NVIDIA GPUs, though specific hardware details were not fully disclosed. The company also introduced a technique called "structured prompting" to improve the model's ability to follow instructions and perform tasks like question answering and reasoning.
Capabilities and Performance
Jurassic-1 demonstrated strong performance on a variety of benchmarks, including SuperGLUE and other NLP tasks, often matching or exceeding GPT-3's results on similar evaluations. The model's larger vocabulary and training data allowed it to handle rare words and multilingual text more effectively. In addition to standard text generation, Jurassic-1 could perform tasks such as named entity recognition, sentiment analysis, and text classification without fine-tuning, using few-shot learning.
The model also supported a feature called "zero-shot" reasoning, where it could answer questions or solve problems without prior examples. AI21 Labs released a public playground and API, allowing developers to integrate Jurassic-1 into applications. The API offered options for temperature control, top-p sampling, and other generation parameters.
Comparison with Contemporaries
At its release, Jurassic-1 was positioned as a direct competitor to OpenAI's GPT-3, which had been released in 2020. While GPT-3 had 175 billion parameters, Jurassic-1 Jumbo had 178 billion, making it slightly larger. However, Jurassic-1's tokenizer used a larger vocabulary, which reduced the number of tokens needed for a given text, potentially improving inference speed and cost. AI21 Labs also highlighted the model's ability to handle longer contexts, with a maximum sequence length of 2,048 tokens, similar to GPT-3.
Unlike Anthropic's later models or Google DeepMind's efforts, Jurassic-1 was not focused on reinforcement learning from human feedback (RLHF) at launch, but rather on supervised learning and few-shot prompting. This made it more similar to the original GPT-3 in approach.
Impact and Legacy
Jurassic-1 was one of the first major LLMs from a company other than OpenAI, helping to establish AI21 Labs as a significant player in the generative AI space. It was used in various commercial applications, including writing assistance, customer service chatbots, and content generation. The model's release also contributed to the broader trend of increasing model sizes and the development of large language models as a service.
Subsequent models from AI21 Labs, such as Jurassic-2, built upon the architecture and lessons learned from Jurassic-1, incorporating improvements in training data and alignment. The company continued to offer Jurassic-1 through its API for some time, though it was eventually deprecated in favor of newer models. Jurassic-1 remains a notable milestone in the history of artificial intelligence research, demonstrating that competitive LLMs could be developed outside of the largest tech companies.
Reception and Criticism
Jurassic-1 received positive reviews for its performance and accessibility, with many developers praising the API's ease of use and the model's ability to generate coherent and contextually relevant text. However, like other LLMs, it faced criticism regarding potential biases in its training data and the environmental impact of training such large models. AI21 Labs acknowledged these concerns and published documentation on responsible use, but did not release the model's weights publicly, limiting independent research.
Some researchers noted that Jurassic-1's performance on certain reasoning tasks was not significantly better than smaller models, highlighting the limitations of scale alone. Despite these critiques, the model's release helped to spur competition in the LLM market, leading to rapid advancements in the field.