ImageNet-Sketch is a benchmark dataset introduced in 2019 to evaluate the cross-domain generalization of machine learning models. It contains 50,000 grayscale sketch images, with 50 images for each of the 1,000 classes in the original ImageNet dataset. The sketches are collected from Google's QuickDraw project and other sources, and they represent a stylized, abstract depiction of objects, differing significantly from natural photographs. This dataset is widely used in computer vision research to test how well models trained on natural images can recognize sketches, a task that requires robustness to domain shift.
The dataset was created by researchers at MIT and Google, and it was first presented in the paper 'Learning Robust Representations by Projecting Supervisory Signals' by Haotian Wang, Weichao Qiu, and others. The primary motivation was to provide a challenging evaluation set that goes beyond standard test sets, which often share the same distribution as training data. By using sketches, which are visually distinct from photos, ImageNet-Sketch forces models to rely on high-level semantic features rather than low-level texture or color cues.
Construction and Composition
ImageNet-Sketch comprises 50,000 images, with exactly 50 sketches per class across all 1,000 ImageNet categories. The sketches are sourced from the QuickDraw dataset, which contains millions of user-drawn doodles, and are filtered to match ImageNet class labels. Each sketch is a black-and-white line drawing, typically simple and abstract, capturing the essential shape of an object. The dataset intentionally avoids color and texture information, making it a pure test of shape-based recognition.
The construction process involved matching QuickDraw categories to ImageNet synsets, and then selecting a subset of sketches that were deemed representative. The final dataset is publicly available for download, and it has become a standard benchmark in the field. Researchers often use it in conjunction with other datasets, such as ImageNet-C or ImageNet-A, to assess robustness to various types of distribution shift.
Use in Model Evaluation
ImageNet-Sketch is primarily used to evaluate the cross-domain generalization of Deep learning models, particularly convolutional neural networks and more recent Transformer (architecture)-based architectures. When a model is trained on ImageNet (natural images) and tested on ImageNet-Sketch, the performance drop indicates how well the model has learned transferable features. Models that rely heavily on texture or background cues tend to perform poorly, while those that capture shape and structure perform better.
For example, ResNet models and vision transformers have been evaluated on this benchmark. Studies have shown that vision transformers often outperform convolutional networks on sketch recognition, possibly due to their global attention mechanisms. The dataset is also used in Data Augmentation research, where techniques like style transfer or adversarial training are tested for their ability to improve robustness to domain shift.
Relation to Other Benchmarks
ImageNet-Sketch is part of a family of robustness benchmarks that include ImageNet-C (corruptions), ImageNet-A (adversarial examples), and ImageNet-R (renditions). While ImageNet-C applies synthetic corruptions to natural images, ImageNet-Sketch provides a completely different visual style. This makes it a complementary test for evaluating model generalization. The dataset is often used in leaderboards and research papers to report robustness metrics alongside accuracy on the original ImageNet validation set.
The dataset has also been used to study the phenomenon of 'texture bias' in neural networks, where models tend to rely on local texture patterns rather than global shape. By presenting sketches that lack texture, ImageNet-Sketch exposes this bias and encourages the development of more shape-aware models.
Limitations and Criticisms
One limitation of ImageNet-Sketch is that the sketches are not uniformly distributed across all classes; some classes may have more or fewer than 50 images, although the dataset is designed to be balanced. Additionally, the sketches are relatively simple and may not capture the full complexity of real-world sketches, which could limit the generalizability of the findings. Some researchers have also noted that the dataset is biased towards classes that are easy to sketch, such as animals and common objects, while more abstract classes may be underrepresented.
Despite these limitations, ImageNet-Sketch remains a valuable resource for evaluating cross-domain performance. It is widely cited in the computer vision literature and continues to be used in studies on robustness, transfer learning, and model interpretability.