OpenWebText is an open-source dataset of web pages extracted from Reddit submissions, created to replicate the private WebText corpus used to train OpenAI's GPT-2 language model. It provides a large, diverse text corpus for training and evaluating large language models.
Background and Motivation
OpenWebText was introduced in 2019 by Aaron Gokaslan and Vanya Cohen, researchers at the time affiliated with the University of Maryland and Johns Hopkins University, respectively. The motivation stemmed from OpenAI's decision not to release the full WebText dataset, which was used to train GPT-2. WebText consisted of outbound links from Reddit posts that received at least 3 karma, totaling approximately 40GB of text. To enable reproducibility and further research, Gokaslan and Cohen created OpenWebText by scraping all outbound links from Reddit submissions from 2005 through April 2018, applying the same karma threshold, and filtering for English-language content. The resulting dataset contains over 8 million documents, totaling roughly 38GB of text, closely mirroring the size and diversity of the original WebText.
Dataset Characteristics
OpenWebText is a large-scale, unstructured text corpus. Unlike curated datasets such as Wikipedia or books, it captures a wide variety of writing styles, topics, and formats, including news articles, blog posts, forum discussions, and personal websites. This diversity makes it valuable for training general-purpose language models. The dataset is provided as a collection of text files, with each document separated by a newline. It is commonly used in the research community as a benchmark for evaluating model performance on next-word prediction and other language modeling tasks. However, because it is derived from public web content, it inherits issues such as duplicate content, low-quality text, and potential biases present in online discourse.
Usage in Language Model Training
OpenWebText has been widely adopted in the field of natural language processing (NLP). It serves as a standard training corpus for many open-source language models, including GPT-2 variants and other transformer-based architectures. For example, Hugging Face provides a preprocessed version of OpenWebText that is commonly used for fine-tuning and evaluation. Researchers often use it to compare model performance against GPT-2, as it provides a comparable training distribution. The dataset has also been used in studies on data curation, model scaling, and the effects of training data on model behavior. Its availability has facilitated reproducible research in large language model development, particularly in the context of generative AI.
Impact and Limitations
OpenWebText has had a significant impact on the open-source AI community by democratizing access to a large-scale web corpus. It has enabled researchers without access to proprietary datasets to train and evaluate models of similar scale to GPT-2. However, the dataset has limitations. It is not as clean as curated corpora, and it contains a substantial amount of boilerplate text, navigation elements, and other noise. Additionally, the scraping methodology based on Reddit karma introduces a selection bias, favoring content that appeals to Reddit users. The dataset also lacks explicit licensing information for many of the included web pages, raising copyright concerns. Despite these issues, OpenWebText remains a foundational resource in the development of large language models and continues to be referenced in academic literature.
Related Datasets and Evolution
OpenWebText is part of a broader ecosystem of web-scale text datasets used in NLP. Other notable examples include Common Crawl, which is a massive crawl of the web, and The Pile, a curated collection of diverse sources. More recent datasets, such as C4 (Colossal Clean Crawled Corpus) and RefinedWeb, have been developed to address some of the noise and quality issues present in raw web crawls. These newer datasets employ more sophisticated filtering and deduplication techniques. Nevertheless, OpenWebText remains relevant as a historical benchmark and a baseline for evaluating data curation methods. Its creation also highlighted the importance of open data in the field of artificial intelligence, influencing subsequent efforts to release training corpora for public use.
See Also
- Large language model
- GPT-2
- Common Crawl
- The Pile
- Data curation