# WikiText WikiText is a large-scale language modeling dataset derived from Wikipedia articles, widely used for training and evaluating neural network models in natural language processing. WikiText is a collection of English text datasets extracted from verified Wikipedia articles, designed for training and benchmarking language models. It was introduced to provide a larger, more realistic corpus than earlier datasets like Penn Treebank, with a focus on preserving the original article structure, including headings, lists, and links. The dataset is commonly used in research on [machine learning](https://www.wikiprompt.org/wiki/machine-learning) and [deep learning](https://www.wikiprompt.org/wiki/deep-learning) for tasks such as next-word prediction and model evaluation. The primary versions are WikiText-2 and WikiText-103, containing 2 million and 103 million tokens respectively. These datasets are curated to exclude tables and lists, and they retain the raw text with special tokens for unknown words, making them suitable for testing models' ability to handle a large vocabulary and long-term dependencies. WikiText has become a standard benchmark in the field, cited in numerous papers on [neural networks](https://www.wikiprompt.org/wiki/neural-network) and [large language models](https://www.wikiprompt.org/wiki/large-language-model). ## Dataset Construction WikiText was created by Salesforce Research in 2016, with the goal of providing a more challenging corpus than previous options. The extraction process involved taking articles from Wikipedia that were rated as 'Good' or 'Featured' by the Wikipedia community, ensuring high quality and consistency. The text was then cleaned to remove markup, but the original casing, punctuation, and numbers were preserved, which is a departure from earlier datasets that often lowercased everything. A key feature is the use of a 267,735-word vocabulary for WikiText-103, which includes all words appearing at least three times in the training set. Rare words are replaced with an '' token. This large vocabulary forces models to learn representations for many words, testing their ability to generalize. The dataset is split into training, validation, and test sets, with the test set containing articles from a different time period to avoid leakage. ## Applications in Language Modeling WikiText is primarily used for evaluating language models, which predict the probability of a sequence of words. Researchers use it to measure perplexity, a metric indicating how well a model predicts the next word. Lower perplexity scores are better, and WikiText-103 has become a standard for comparing [transformer](https://www.wikiprompt.org/wiki/transformer)-based models, such as those developed by [openai](https://www.wikiprompt.org/wiki/openai) and [google-deepmind](https://www.wikiprompt.org/wiki/google-deepmind). For example, models like GPT-2 and BERT have been evaluated on WikiText-103, with reported perplexity scores improving over time. The dataset's long articles, averaging around 3,000 words, test a model's ability to handle long-range context, which is crucial for tasks like document summarization and question answering. This makes WikiText a valuable resource for advancing [artificial intelligence](https://www.wikiprompt.org/wiki/artificial-intelligence) research. ## Comparison with Other Datasets Before WikiText, the standard benchmark was the Penn Treebank (PTB), which contains only about 1 million tokens from Wall Street Journal articles. PTB is smaller and has a restricted vocabulary, making it easier for models to achieve low perplexity. WikiText offers a more realistic challenge by using a larger vocabulary and longer documents, which better reflects real-world text. Another dataset, the One Billion Word Benchmark, is larger but consists of shuffled sentences, losing document structure. WikiText retains the original article order, allowing models to learn from coherent narratives. This structural difference is significant for training models that need to understand context across sentences and paragraphs, a key capability for [generative AI](https://www.wikiprompt.org/wiki/generative-ai) systems. ## Limitations and Criticisms Despite its popularity, WikiText has limitations. The dataset is derived from Wikipedia, which has a formal, encyclopedic style, so it may not represent informal or conversational language. Additionally, the cleaning process removes tables and lists, which can be informative for certain tasks. Some researchers have noted that the '' token handling can bias evaluation, as models may rely on it rather than learning rare words. Another criticism is that WikiText-103, while large, is still smaller than modern web-scale datasets used for training [large language models](https://www.wikiprompt.org/wiki/large-language-model) like those from [anthropic](https://www.wikiprompt.org/wiki/anthropic) or [microsoft](https://www.wikiprompt.org/wiki/microsoft). As of 2025, many models are trained on trillions of tokens, making WikiText more of a benchmark for academic research than a training source for production systems. Nevertheless, it remains a standard for reproducible comparisons in the field. ## Future Directions The development of WikiText has influenced the creation of other datasets, such as The Pile and RedPajama, which combine multiple sources. These newer datasets aim to address WikiText's limitations by including diverse text types and larger scales. However, WikiText's simplicity and ease of use ensure its continued relevance in education and research, particularly for those studying [deep learning](https://www.wikiprompt.org/wiki/deep-learning) fundamentals. Researchers are also exploring ways to adapt WikiText for multilingual and multimodal tasks, though the original version is English-only. As [AI](https://www.wikiprompt.org/wiki/artificial-intelligence) models evolve, the need for high-quality, structured datasets remains, and WikiText serves as a foundational example of how to construct such resources. Its legacy is evident in the many papers that cite it as a baseline, ensuring its place in the history of [machine learning](https://www.wikiprompt.org/wiki/machine-learning) development. ## See Also - [Neural network](https://www.wikiprompt.org/wiki/neural-network) - [Transformer](https://www.wikiprompt.org/wiki/transformer) - [Large language model](https://www.wikiprompt.org/wiki/large-language-model) - [Generative AI](https://www.wikiprompt.org/wiki/generative-ai) --- Source: https://www.wikiprompt.org/wiki/wikitext License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/) Last updated: 2026-09-07T02:34:56.62918+00:00