BigScience is an open research collaboration that developed BLOOM, a large language model released in 2022. The initiative brought together hundreds of researchers from various institutions to build a publicly accessible model, contrasting with proprietary efforts. Its goal was to advance Artificial intelligence research through openness, reproducibility, and community involvement.
The project was launched in 2021, coordinated by Hugging Face, and received support from organizations including the French government and various academic partners. BigScience aimed to address concerns about the concentration of AI development in a few large companies by creating a collaborative, transparent alternative. The resulting model, BLOOM, was trained on a multilingual dataset and made available under an open license.
Origins and Goals
BigScience emerged from discussions within the AI community about the need for more inclusive and ethical development of large language models. The collaboration was formally announced in May 2021, with the intention of pooling expertise and computational resources. Key objectives included producing a state-of-the-art model, documenting the entire process, and establishing best practices for responsible AI research.
The initiative was notable for its scale and diversity, involving over 1,000 contributors from more than 70 countries. Participants included researchers from academia, industry, and civil society, reflecting a broad range of perspectives. BigScience sought to demonstrate that collaborative, open research could rival the achievements of well-funded corporate labs.
BLOOM Model
BLOOM (BigScience Large Open-science Open-access Multilingual) is a Transformer (architecture)-based neural network with 176 billion parameters. It was trained on a dataset called ROOTS, which comprised over 1.6 terabytes of text in 46 natural languages and 13 programming languages. The training used the Jean Zay supercomputer in France, provided through a grant from the French government.
BLOOM was designed to support text generation and understanding across multiple languages, including low-resource ones. Its architecture incorporates Multi-Head Attention and other standard transformer components, but with modifications for efficiency and stability. The model was released in July 2022 under the Responsible AI License, which includes usage restrictions to prevent harmful applications.
Training Data and Methodology
The ROOTS dataset was assembled through a participatory process, with volunteers curating and filtering web crawls, academic papers, and other sources. Emphasis was placed on data quality and provenance, with detailed documentation of the collection and preprocessing steps. This transparency was a departure from many proprietary models, which often keep training data confidential.
Training BLOOM required significant computational resources, utilizing thousands of GPUs over several months. The collaboration employed techniques such as Gradient Clipping and Layer Normalization to ensure stable training. The project also published detailed logs and analyses, enabling others to replicate or build upon the work.
Governance and Ethics
BigScience established a governance structure that included working groups on ethics, legal issues, and community engagement. These groups developed guidelines for responsible data collection, model deployment, and potential misuse. The collaboration also conducted ethical reviews and sought input from affected communities.
One notable outcome was the creation of the Responsible AI License, which restricts use of BLOOM in high-risk domains such as surveillance or discrimination. This license was designed to balance openness with accountability, a novel approach at the time. The project also published a model card detailing capabilities, limitations, and intended uses.
Impact and Reception
BLOOM was widely praised for its accessibility and multilingual capabilities, particularly for languages often neglected by commercial models. It provided a benchmark for open research, showing that collaborative efforts could produce competitive results. The project also influenced subsequent initiatives, such as the creation of other open models and increased attention to data governance.
Critics noted that BLOOM's performance lagged behind some proprietary models on certain tasks, and that its size made deployment challenging for smaller organizations. Nevertheless, the release of BLOOM contributed to a broader movement toward open-source AI, with many later models drawing inspiration from its approach.
Legacy and Future Directions
BigScience concluded its main phase with the release of BLOOM, but its legacy persists in ongoing open research efforts. The collaboration demonstrated the value of diverse participation and transparent methodologies. It also highlighted the importance of ethical considerations in AI development, influencing policy discussions and industry practices.
As of 2025, the BigScience model and dataset remain available for research and commercial use under the specified license. The project's documentation and code continue to serve as resources for those studying large-scale model training. Future open collaborations may build on BigScience's framework to address emerging challenges in AI.
See Also
References
This article is based on publicly available information about the BigScience project and its outputs.