Yelp Full is a widely used benchmark dataset in natural language processing (NLP) for the task of review rating classification. It consists of Yelp reviews labeled with a star rating from 1 to 5, and the goal is to predict the rating given the review text. The dataset is commonly employed to evaluate the performance of text classification models, including traditional machine learning approaches and modern deep learning architectures.
The dataset was introduced as part of the "Character-level Convolutional Networks for Text Classification" paper by Xiang Zhang et al. in 2015. It is derived from the larger Yelp dataset, which contains millions of user reviews of local businesses. The Yelp Full version is constructed by taking a subset of reviews, ensuring a balanced distribution across the five rating classes. The training set contains 650,000 samples, and the test set contains 50,000 samples. Each review is a raw text string, and the label is the corresponding star rating (1 to 5).
Dataset Construction
The Yelp Full dataset is created from the Yelp Dataset Challenge, which provides a large collection of user reviews. To build the balanced subset, the authors randomly sampled 130,000 reviews per rating class for training, totaling 650,000, and 10,000 per class for testing, totaling 50,000. This balanced design ensures that accuracy is a meaningful metric, as random guessing would yield 20% accuracy. The reviews are not preprocessed beyond basic tokenization, preserving the original text for character-level or word-level modeling.
Usage in Research
Yelp Full is frequently used as a standard benchmark for text classification. It appears in research papers evaluating convolutional neural networks (CNNs), recurrent neural networks (RNNs), and transformer-based models. For example, it is one of the datasets in the widely cited benchmark suite by Zhang et al., which also includes AG News, DBPedia, and Amazon reviews. The dataset is particularly useful for studying the impact of model architecture, embedding techniques, and training strategies on classification accuracy.
In recent years, Yelp Full has been used to test large language models (LLMs) and transfer learning approaches. Models such as BERT and its variants achieve high accuracy on this task, often exceeding 95% on the test set. However, the dataset remains challenging for models that do not leverage pre-trained representations, making it a useful baseline for comparing different NLP methodologies.
Related Tasks and Variants
Yelp Full is often contrasted with the Yelp Polarity dataset, which is a binary classification version where reviews with ratings 1-2 are labeled negative and 4-5 are labeled positive (rating 3 is excluded). The full version is more granular and thus more challenging. Researchers sometimes use Yelp Full to evaluate regression approaches or ordinal classification methods, since the ratings have a natural order. The dataset is also used in multi-task learning settings, where models predict both rating and other attributes such as review helpfulness.
Evaluation Metrics
Accuracy is the primary evaluation metric for Yelp Full, as the classes are balanced. Some studies also report F1-score, precision, and recall, particularly when analyzing per-class performance. Since the dataset is balanced, accuracy provides a straightforward comparison across models. The state-of-the-art results on Yelp Full have improved significantly since its introduction, from around 60% accuracy with character-level CNNs to over 96% with fine-tuned transformer models.
See Also
- Machine learning
- Deep learning
- Neural network
- Large language model
- Transformer (architecture)
- Natural language processing
References
- Zhang, X., Zhao, J., & LeCun, Y. (2015). Character-level Convolutional Networks for Text Classification. Advances in Neural Information Processing Systems.
- Yelp Dataset Challenge (https://www.yelp.com/dataset)