Dolly is an instruction-tuned Large language model developed by Databricks, first released in March 2023. It demonstrated that a relatively small model, fine-tuned on a modest dataset, could achieve competent instruction-following behavior, challenging the prevailing assumption that massive scale and extensive proprietary data were necessary for such capabilities. The project aimed to make LLM customization accessible to enterprises by showing that fine-tuning could be done on a single machine in a short time.
The initial version, Dolly 1.0, was based on EleutherAI's GPT-J 6B model and fine-tuned on a dataset of roughly 15,000 instruction-response pairs generated by Databricks employees. This was followed by Dolly 2.0 in April 2023, which used the same base model but introduced a larger, open-source dataset of 15,000 pairs, released under a permissive license. Dolly 2.0 became notable for being one of the first open-source instruction-tuned models with a fully open training dataset, allowing researchers to replicate and build upon the work.
Technical Approach
Dolly's training employed a technique known as instruction tuning, where a pre-trained language model is fine-tuned on examples of instructions and corresponding responses. This process aligns the model's behavior with user intent, enabling it to follow prompts across various tasks such as summarization, question answering, and text generation. The fine-tuning used a standard supervised learning objective, optimizing the model to predict the response tokens given the instruction. Databricks utilized its own Machine learning platform to manage the training pipeline, but the model itself was designed to be run on commodity hardware, including a single Nvidia A10 GPU for inference.
The base model, GPT-J, is a Transformer (architecture)-based architecture with 6 billion parameters, trained on the Pile dataset. Dolly's fine-tuning did not alter the base architecture but adjusted the weights to specialize in instruction following. This approach contrasted with larger models like those from OpenAI or Anthropic, which often require massive computational resources and proprietary training data.
Release and Impact
Dolly was released under the Apache 2.0 license, making it freely available for commercial and research use. The accompanying dataset, dubbed 'databricks-dolly-15k', was also open-sourced, a rarity at the time. This openness allowed the Machine learning community to study the effects of instruction tuning and to create derivative models. Dolly's release sparked discussions about the democratization of AI, as it showed that organizations without vast resources could build capable assistants.
However, Dolly's performance was not on par with state-of-the-art models like GPT-3.5 or Claude, and it exhibited limitations in complex reasoning and factual accuracy. Nevertheless, it served as a proof of concept for efficient fine-tuning, influencing later open-source efforts such as Alpaca and Vicuna. Databricks positioned Dolly as a tool for enterprises to create custom AI assistants using their own data, integrating with their lakehouse architecture.
Reception and Legacy
Dolly received significant media attention upon release, with coverage in technology press highlighting its low training cost (reported at under $30 for compute) and its challenge to the narrative that only tech giants could develop LLMs. The model was praised for its transparency, but some critics noted that its capabilities were limited compared to commercial offerings. Over time, Dolly was superseded by more powerful open models, but its dataset and methodology continued to be used in research.
Databricks later integrated Dolly into its platform, allowing customers to fine-tune and deploy the model for specific use cases. The project also contributed to the broader trend of open-source LLM development, encouraging further experimentation with instruction tuning on smaller models. As of 2025, Dolly is considered a historical milestone in the evolution of accessible AI, though it is no longer actively developed.
See Also
- Generative AI
- Deep learning
- Neural network
- Amazon Web Services (for cloud-based LLM deployment)
References
(Note: This article is based on publicly available information and does not cite external sources.)