BLOOM (BigScience Large Open-science Open-access Multilingual Language Model) is an open-access Large language model released in 2022. It was developed by the BigScience initiative, a volunteer-driven research effort aimed at providing a transparently-created alternative to proprietary AI models. With 176 billion parameters, BLOOM is a transformer-based autoregressive model designed to generate text in 46 natural languages and 13 programming languages. The model is distributed under the project's "Responsible AI License."
Development
BLOOM is the main outcome of the BigScience initiative, a one-year-long research workshop coordinated by Hugging Face with funding from the French government. The project involved several hundred volunteer researchers and engineers from academia and the private sector. The model was trained between March and July 2022 on the Jean Zay public supercomputer in France, managed by GENCI and IDRIS (CNRS). Unlike earlier models such as GPT-3, BLOOM was explicitly trained to be multilingual, covering languages from diverse families including English, French, Spanish, Chinese, Arabic, Hindi, and many others.
The source code for BLOOM is released under the Apache 2.0 license, while the model's parameters are released under the BigScience Responsible AI License (RAIL). This license grants open access and reuse rights but includes usage restrictions, such as prohibiting use in harmful applications. The open nature of BLOOM contrasts with the proprietary models from companies like OpenAI and Google DeepMind, making it a significant resource for researchers and developers seeking transparency in AI.
Training Data and Architecture
BLOOM's training corpus, named ROOTS, combines data extracted from the then-latest version of the web-based OSCAR corpus (38% of ROOTS) with newly collected data from a manually selected and documented list of language data sources. In total, the model was trained on approximately 366 billion tokens (1.6 TB of text). The architecture is based on the Transformer (architecture) model, specifically an autoregressive decoder-only design, similar to GPT-3 but with enhancements for multilingual performance. The training utilized the open-source libraries DeepSpeed and Megatron, which enable efficient distributed training across thousands of GPUs.
BLOOM's ability to handle 46 natural languages and 13 programming languages makes it versatile for tasks such as translation, summarization, and code generation. Its multilingual training is a key differentiator from many English-centric models, allowing it to serve diverse global communities.
Variants and Applications
Following the release of BLOOM, BigScience introduced xP3, a multilingual dataset for supervised learning, and BLOOMZ, a variant of BLOOM fine-tuned on xP3 to follow instructions. BLOOMZ improves the model's ability to perform tasks based on natural language prompts, enhancing its usability in conversational AI and other applications.
BLOOM has been used in chatbots such as BLOOMChat and HuggingChat, leveraging its multilingual capabilities to provide responses in multiple languages. Its open-access nature has also made it a popular choice for academic research and experimentation in Machine learning and Artificial intelligence fields.
Impact and Significance
BLOOM represents a milestone in the democratization of AI, offering a high-capacity model that is freely available for research and commercial use under its license. It was one of the first large-scale multilingual models to be fully open-sourced, setting a precedent for future initiatives. The project's emphasis on transparency - from data collection to training procedures - provides a valuable case study for the AI community.
Compared to proprietary models like those from OpenAI or Anthropic, BLOOM's open weights allow researchers to inspect and fine-tune the model without restrictions, fostering innovation in areas such as low-resource language processing and AI safety. The model's development also highlighted the potential of collaborative, volunteer-driven research, as opposed to corporate-led efforts.
External Links
- BigScience project on HuggingFace (available at huggingface.co/bigscience)
References
- BigScience project on HuggingFace
- Wikipedia contributors. "BLOOM (language model)." Wikipedia, The Free Encyclopedia.