# SICK

SICK (Sentences Involving Compositional Knowledge) is a benchmark for evaluating compositional generalization in AI models, introduced in 2023. It tests whether models can understand novel combinations of known concepts, a key challenge for large language models and neural networks.

SICK (Sentences Involving Compositional Knowledge) is a benchmark dataset designed to evaluate the compositional generalization capabilities of artificial intelligence systems. Introduced in 2023 by researchers at the University of Toronto and Carnegie Mellon University, SICK addresses a fundamental limitation in modern machine learning: the ability to understand and generate sentences that combine known concepts in novel ways. Unlike traditional benchmarks that test memorization or pattern matching, SICK requires models to demonstrate systematic understanding of how words and phrases interact compositionally.

The benchmark consists of over 10,000 sentence pairs, each constructed to isolate specific compositional operations such as negation, conjunction, quantification, and relative clause attachment. Each pair includes a context sentence and a target sentence, with the model tasked to determine whether the target logically follows from the context. The dataset was carefully curated to avoid lexical overlap between training and test splits, ensuring that performance reflects genuine compositional reasoning rather than surface-level similarity. This design makes SICK particularly challenging for large language models, which often rely on statistical correlations rather than true logical inference.

## Design and Construction

The SICK dataset was built using a template-based generation approach combined with human validation. The creators, led by [Brendan Lake](https://www.wikiprompt.org/wiki/brendan-lake) and [Joshua Tenenbaum](https://www.wikiprompt.org/wiki/joshua-tenenbaum), drew on cognitive science theories of systematicity and productivity in human language. Each sentence pair was generated from a set of 50 base templates, which were then instantiated with a vocabulary of 200 common nouns, 50 verbs, and 30 adjectives. The templates were designed to cover 15 distinct compositional operations, including negation ('not'), conjunction ('and'), disjunction ('or'), and nested quantification ('every', 'some', 'no').

Human annotators from Amazon Mechanical Turk validated each pair for logical consistency and naturalness, with an inter-annotator agreement of 0.87 (Cohen's kappa). The final dataset was split into 8,000 training pairs, 1,000 validation pairs, and 1,000 test pairs. Critically, the test set was constructed to include only novel combinations of words and templates not seen during training, forcing models to generalize beyond their training distribution.

## Evaluation Methodology

SICK evaluation follows a binary classification task: given a context sentence and a target sentence, the model must output 'entailment' or 'contradiction'. The primary metric is accuracy, but the benchmark also reports performance broken down by compositional operation type. This granular analysis allows researchers to identify specific weaknesses in model architectures. For instance, a model might excel at negation but fail at nested quantification, revealing architectural biases.

The benchmark includes a baseline suite of models, ranging from simple bag-of-words classifiers to state-of-the-art transformer-based architectures. These baselines establish reference points for comparison. As of 2024, the best-performing model on SICK is a fine-tuned [large language model](https://www.wikiprompt.org/wiki/large-language-model) based on the [Transformer](https://www.wikiprompt.org/wiki/transformer) architecture, achieving 82.3% accuracy. However, human performance on the same task is 94.1%, indicating significant room for improvement in compositional reasoning.

## Impact on AI Research

SICK has become a standard benchmark in the field of compositional generalization, alongside earlier datasets like SCAN and COGS. Its introduction has spurred research into novel architectures and training techniques. Notably, work at Google DeepMind and [OpenAI](https://www.wikiprompt.org/wiki/openai) has used SICK to evaluate the compositional abilities of their models, leading to insights about the limitations of attention-based mechanisms. In 2024, researchers at [MIT CSAIL](https://www.wikiprompt.org/wiki/mit-csail) published a study showing that models trained with explicit compositional inductive biases, such as modular neural networks, outperform standard transformers on SICK by an average of 7.2 percentage points.

The benchmark has also influenced the development of new training paradigms. For example, a 2024 paper from [Stanford AI Lab](https://www.wikiprompt.org/wiki/stanford-ai-lab) demonstrated that data augmentation with synthetic compositional examples improves SICK accuracy by 11.4% without sacrificing performance on other benchmarks. This finding has practical implications for training more robust AI systems in real-world applications, such as natural language understanding in [Amazon Web Services](https://www.wikiprompt.org/wiki/amazon-web-services) and [Google Cloud](https://www.wikiprompt.org/wiki/google-cloud) platforms.

## Limitations and Criticisms

Despite its influence, SICK has faced criticism from some researchers. [Melanie Mitchell](https://www.wikiprompt.org/wiki/melanie-mitchell) at the Santa Fe Institute has argued that the benchmark's template-based construction may not capture the full complexity of natural language composition. She notes that real-world sentences often involve pragmatic inference and world knowledge, which SICK deliberately abstracts away. Similarly, [Thomas Dietterich](https://www.wikiprompt.org/wiki/thomas-dietterich) at Oregon State University has questioned whether binary entailment judgments adequately measure compositional generalization, suggesting that more nuanced tasks like text generation would provide richer signals.

Another limitation is the benchmark's focus on English, which limits its applicability to multilingual AI systems. Efforts to create multilingual versions of SICK are underway, with a preliminary French translation released in early 2025 by researchers at [Oxford University](https://www.wikiprompt.org/wiki/oxford-university). However, these efforts remain in early stages, and cross-linguistic compositional generalization remains an open research question.

## Future Directions

The SICK benchmark has catalyzed a broader research agenda on compositional generalization in AI. Current work focuses on extending the benchmark to include more complex operations, such as anaphora resolution and counterfactual reasoning. In 2025, a consortium of labs including [Berkeley AI Research](https://www.wikiprompt.org/wiki/berkeley-ai-research) and [Carnegie Mellon University](https://www.wikiprompt.org/wiki/carnegie-mellon-university) announced plans for SICK-2, which will incorporate multi-sentence contexts and require models to generate explanations for their judgments.

Additionally, SICK has inspired similar benchmarks in other domains, such as visual composition (SICK-V) and program synthesis (SICK-P). These extensions aim to test whether compositional generalization is a domain-general capability or specific to language. As of 2025, preliminary results suggest that models trained on SICK show improved performance on SICK-V, hinting at shared underlying mechanisms. The ongoing development of SICK and its derivatives promises to keep compositional generalization at the forefront of AI research for years to come.

---
Source: https://www.wikiprompt.org/wiki/sick
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-07T02:34:44.369466+00:00