ARC-AGI, short for the Abstraction and Reasoning Corpus for Artificial General Intelligence, is a AI benchmark created by Francois Chollet, the creator of the Keras deep learning framework, and first released in 2019. It consists of visual grid puzzles in which a solver must infer an abstract transformation rule from a small number of example input-output grid pairs and then apply that rule correctly to a new test grid.
Purpose
ARC-AGI was designed explicitly as a response to the observation that many AI benchmarks, however difficult, could be substantially solved through large-scale pattern matching or memorization rather than genuine reasoning or generalization. Chollet argued, in an accompanying paper "On the Measure of Intelligence," that intelligence should be measured by skill-acquisition efficiency on novel tasks rather than by raw performance on any fixed, learnable skill, and designed ARC-AGI's puzzles to each require a distinct, generalizable abstraction, drawing on concepts humans acquire early such as symmetry, counting, containment, and object permanence, deliberately chosen to be difficult to shortcut with memorized patterns.
Difficulty for AI systems
For years after its release, ARC-AGI proved strikingly resistant to both classical machine learning and early large language models, with even capable GPT-4-class models solving only a small fraction of the public evaluation set, while most humans, including children, solve the majority of puzzles without special training. This gap between strong performance on conventional benchmarks like MMLU and weak performance on ARC-AGI was frequently cited as evidence that scaling alone, without qualitatively different reasoning capability, was insufficient to reach humanlike generalization.
Progress with reasoning models
The gap narrowed significantly with the emergence of reasoning models that use extended test-time compute and chain-of-thought-style search, most notably OpenAI's o1 and its successors, which posted substantially higher ARC-AGI scores than prior non-reasoning models, though often at significantly higher inference cost per puzzle. This progress was taken by some as evidence that search and deliberation at inference time, rather than pretraining scale alone, was a meaningful path toward more general reasoning, while others cautioned that gains partly reflected models learning ARC-AGI-like puzzle patterns from public leaderboard data rather than a fully general reasoning improvement.
Format and prize
ARC-AGI puzzles are intentionally kept partially private, with a held-out evaluation set not published, specifically to limit the contamination problems that undermined earlier static benchmarks. An associated public competition, the ARC Prize, has offered cash prizes for systems that achieve strong performance on the held-out set, drawing both academic and industry participants and periodically releasing harder successor versions, including ARC-AGI-2, as earlier versions approached saturation.
Significance
ARC-AGI is frequently invoked in debates about AGI timelines and about whether current deep learning approaches are sufficient to reach general intelligence, precisely because its design goal, resisting memorization while remaining easy for humans, makes it a rare benchmark where large gaps between human and AI performance persisted well into the era of highly capable large language models. It remains one of the most closely watched indicators of genuine reasoning progress, distinct from benchmarks more susceptible to scale and data alone.