Common Crawl is a nonprofit organization, and the name of its regularly updated, freely available archive of petabytes of web crawl data, collected by systematically crawling billions of web pages and releasing the raw HTML, extracted text, and metadata for public download. Founded in 2007 by Gil Elbaz, Common Crawl predates the modern era of large language models by more than a decade, but its archive became one of the most important raw materials for LLM pretraining once researchers began training large text models on web-scale data in the late 2010s and 2020s.
Common Crawl releases a new crawl roughly monthly, and by the mid-2020s its cumulative archive spanned hundreds of billions of web pages, making it, alongside licensed and curated datasets, one of the largest single sources of raw text available for AI training.
History and operation
Common Crawl was established as a nonprofit with the stated mission of democratizing access to web-scale data that had previously been available mainly to large search engine companies, providing researchers, startups, and independent developers a resource comparable in scale to what Google or Microsoft could crawl internally. The organization has received funding and support from a number of technology companies and foundations over the years. Its crawler follows links across the public web largely indiscriminately, subject to robots.txt and other standard crawling conventions, meaning the raw archive includes a very wide range of content quality, from high-value reference material to spam, marketing text, and low-quality AI-generated content in more recent crawls.
Role in AI training
Because Common Crawl's raw archive is unfiltered and highly heterogeneous, most AI labs do not train directly on it but instead build derived, filtered datasets on top of it. Well-known examples include the C4 dataset used to train Google's T5 model, the datasets underlying early GPT models such as GPT-2 and GPT-3, and RefinedWeb and FineWeb, filtered and deduplicated Common Crawl derivatives released by Hugging Face-affiliated and other research groups in the 2020s specifically for open foundation model training. These derived datasets apply steps such as deduplication, language identification, quality classifiers trained to mimic curated reference text like Wikipedia, and toxicity or spam filtering, typically retaining only a fraction of the original crawled pages.
Criticism and controversy
Common Crawl's role as a foundational input to commercial AI training drew increasing scrutiny alongside the broader wave of AI copyright lawsuits from 2023 onward, since the archive includes copyrighted news articles, books, and other material scraped without the rights holders' explicit permission, collected under the same web-crawling norms search engines had long relied on but applied to a new and more contentious use case. Some publishers moved to block Common Crawl's crawler, CCBot, via robots.txt after 2023 specifically to prevent their content from flowing into AI training pipelines, a response mirrored across many sites toward AI-related crawlers generally. Common Crawl has stated it complies with such opt-outs and operates as a neutral data-collection service, arguing that responsibility for downstream use lies with the organizations that build training datasets from its archive.
Significance
Common Crawl's continued existence as a free, open alternative to proprietary web indices has been cited by researchers, including at organizations such as EleutherAI and the Allen Institute for AI, as important for enabling non-commercial and open-source AI research to remain viable in a field increasingly dominated by well-funded closed labs with their own private crawling infrastructure.