HumanEval is a code-generation AI benchmark released by OpenAI in 2021 alongside the paper introducing Codex, the model that powered the original GitHub Copilot. It consists of 164 hand-written Python programming problems, each including a function signature, a docstring describing the task, and a set of unit tests used to check the correctness of a generated solution.
Design
Each HumanEval problem asks a model to complete a Python function body given only its signature and a natural-language docstring, similar to how a programmer might describe a small task before writing it. Correctness is judged functionally: a generated solution passes if it satisfies the accompanying unit tests, rather than being compared textually to a single reference answer, which allows for the many different valid ways to implement the same function. Because the problems were written by hand specifically for the benchmark rather than scraped from existing code repositories, HumanEval was designed to reduce the risk that solutions already existed verbatim in a model's training data.
The pass@k metric
HumanEval introduced or popularized the pass@k metric for evaluating code generation, which measures the probability that at least one of k independently sampled solutions from a model passes all unit tests for a given problem. Pass@1 approximates single-attempt accuracy, while pass@100 or higher values estimate how often a correct solution appears somewhere within many samples, useful for understanding a model's capability under different real-world usage patterns, such as allowing multiple retries.
Adoption and progress
HumanEval quickly became a standard reference point for coding capability, reported alongside general evaluation results in release announcements from OpenAI, Anthropic, Google DeepMind, and open-model developers such as Meta AI with its Llama family. Performance rose quickly: early Codex models solved under 30% of problems at pass@1, while by 2023-2024 leading models exceeded 90%, a saturation pattern similar to that seen with MMLU in the broader knowledge domain.
Limitations and successors
HumanEval's problems are relatively short, self-contained, and Python-specific, meaning strong performance does not necessarily indicate a model can handle large, realistic codebases, multi-file projects, or other programming languages. As with many early benchmarks, data contamination became a growing concern, since the problems and community-written solutions circulated widely online and could plausibly appear in later models' pretraining data. These limitations motivated harder and more realistic successors, most notably SWE-bench, which grounds tasks in real GitHub issues and full codebases rather than short isolated functions, and other multi-language extensions of the original HumanEval format.
Legacy
Despite its narrow scope by later standards, HumanEval played an important role in establishing code generation as a measurable, comparable capability across models, and its functional, test-based evaluation approach, rather than textual similarity to a reference solution, influenced the design of subsequent coding benchmarks across the industry.