Zero-shot prompting is a technique in artificial intelligence and machine learning where a model is asked to perform a task without being provided any examples or demonstrations. The model must rely solely on its pre-trained knowledge and the instructions in the prompt to generate the correct output. This contrasts with few-shot prompting, where a few examples are included in the prompt to guide the model. Zero-shot prompting is a fundamental capability of modern large language models (LLMs), enabling them to handle a wide range of tasks with minimal task-specific data.
The concept is rooted in zero-shot learning (ZSL), a problem setup in machine learning where a learner observes samples from classes not seen during training and must predict their class. The name is a play on words based on the earlier concept of one-shot learning in computer vision. Zero-shot methods generally work by associating observed and non-observed classes through auxiliary information that encodes observable distinguishing properties. For example, a model trained to recognize horses but never shown a zebra can still recognize a zebra if it knows that zebras look like striped horses. In the context of prompting, zero-shot prompting leverages the model's internal representations and semantic understanding to generalize to unseen tasks.
Background and history
The first paper on zero-shot learning in natural language processing appeared in 2008 by Chang, Ratinov, Roth, and Srikumar at AAAI'08, under the name "dataless classification." The first paper on zero-shot learning in computer vision appeared at the same conference, under the name "zero-data learning." The term "zero-shot learning" itself first appeared in a 2009 paper by Palatucci, Hinton, Pomerleau, and Mitchell at NIPS'09. This terminology caught on, as a take-off on one-shot learning introduced in computer vision years earlier.
In computer vision, zero-shot learning models learned parameters for seen classes along with their class representations, relying on representational similarity among class labels so that during inference, instances could be classified into new classes. In natural language processing, the key technical direction built on the ability to "understand the labels" - representing labels in the same semantic space as the documents to be classified. This supported classification of a single example without any annotated data, the purest form of zero-shot classification. The original paper used Explicit Semantic Analysis (ESA) representation, but later papers used dense representations. This approach was extended to multilingual domains, fine entity typing, and other problems. Beyond relying solely on representations, the computational approach was extended to depend on transfer from other tasks, such as textual entailment and question answering.
Prerequisite information for zero-shot classes
Zero-shot learning requires auxiliary information about the unseen classes. Several types of auxiliary information exist:
- Learning with attributes: Classes are accompanied by pre-defined structured descriptions. For example, bird descriptions could include "red head" or "long beak." These attributes are often organized in a structured compositional way, and taking that structure into account improves learning. This approach was used mostly in computer vision, with some examples in natural language processing.
- Learning from textual description: Class labels are taken to have a meaning and are often augmented with definitions or free-text natural-language descriptions, such as a Wikipedia description of the class. This has been the key direction in natural language processing.
- Class-class similarity: Classes are embedded in a continuous space. A zero-shot classifier can predict that a sample corresponds to some position in that space, and the nearest embedded class is used as the predicted class, even if no such samples were observed during training.
In the context of zero-shot prompting, the auxiliary information is implicitly encoded in the model's parameters during pre-training. The model has seen vast amounts of text and learned associations between concepts, allowing it to understand instructions and generate appropriate responses without explicit examples.
Generalized zero-shot learning
The standard zero-shot learning setup assumes that at test time, only zero-shot samples (from unseen classes) are given. In generalized zero-shot learning, samples from both new and known classes may appear at test time. This poses challenges because it is difficult to estimate whether a given sample is new or known. Approaches to handle this include:
- A gating module, trained to decide if a given sample comes from a new class or an old one, and at inference time outputs either a hard or soft probabilistic decision.
- A generative module, trained to generate feature representations of unseen classes, allowing a standard classifier to be trained on samples from all classes, seen and unseen.
For zero-shot prompting, generalized zero-shot learning is less relevant, as the model is typically asked to perform a single task without needing to distinguish between seen and unseen classes. However, the underlying challenge of generalization remains important for robust performance.
Domains of application
Zero-shot learning and zero-shot prompting have been applied to a wide range of fields:
- Image classification: Recognizing objects or scenes not seen during training.
- Semantic segmentation: Assigning labels to pixels in an image, including unseen categories.
- Image generation: Generating images of concepts not explicitly trained on.
- Object detection: Detecting objects from unseen classes.
- Natural language processing: Text classification, sentiment analysis, and question answering without task-specific training data.
- Computational biology: Predicting properties of biological entities, such as protein functions, without labeled examples.
- Abstract reasoning: Solving reasoning tasks that require generalization to novel situations.
In the era of large language models, zero-shot prompting has become a standard evaluation method. Models like those developed by OpenAI, Anthropic, and Google DeepMind are often tested on their zero-shot performance across diverse benchmarks. This capability is a direct result of training on massive corpora and the transformer architecture, which enables the model to internalize vast amounts of knowledge and follow instructions.
Challenges and limitations
Despite its power, zero-shot prompting has limitations. The model may produce plausible but incorrect answers, especially for tasks requiring specialized knowledge or reasoning. The quality of the prompt matters significantly; poorly phrased instructions can lead to suboptimal results. Additionally, zero-shot performance is often lower than few-shot or fine-tuned performance, as the model has no explicit examples to calibrate its output. Researchers continue to explore methods to improve zero-shot capabilities, such as better prompt engineering, chain-of-thought reasoning, and instruction tuning.