# SWE-bench

SWE-bench is a benchmark that evaluates a language model's ability to resolve real-world software engineering issues by generating code patches for genuine GitHub issues, widely used to measure agentic coding capability.

SWE-bench, short for Software Engineering Benchmark, is a [benchmark](https://www.wikiprompt.org/wiki/benchmark) introduced by researchers at Princeton in 2023 that measures a language model's ability to resolve real software engineering tasks. Unlike synthetic coding benchmarks such as [HumanEval](https://www.wikiprompt.org/wiki/humaneval), SWE-bench draws its tasks directly from actual GitHub issues and their corresponding merged pull requests across popular open-source Python repositories, asking a model to generate a code patch that resolves the issue and passes the project's existing test suite.

## Design

Each SWE-bench task presents a model with a codebase at a specific commit, an issue description written by a real user or contributor, and the requirement to produce a patch; success is measured by whether the patch causes the relevant tests to pass without breaking existing ones. This makes SWE-bench considerably harder and more realistic than function-level benchmarks, since resolving an issue often requires understanding a large, unfamiliar codebase, identifying the relevant files among many, and making changes that integrate correctly with existing code, rather than writing an isolated function from a clear specification.

## Relationship to agentic coding

Because real-world issue resolution typically requires multiple steps, exploring a codebase, running tests, and iterating on a fix, SWE-bench became closely associated with the rise of [agentic](https://www.wikiprompt.org/wiki/ai-agent) coding systems rather than simple single-shot code generation. Tools such as [GitHub Copilot](https://www.wikiprompt.org/wiki/github-copilot)'s agent modes, [Cursor](https://www.wikiprompt.org/wiki/cursor), and [Claude Code](https://www.wikiprompt.org/wiki/claude-code) are commonly evaluated against SWE-bench or its variants as a proxy for real-world usefulness, and the benchmark's rise tracked the broader shift in the industry from viewing LLM coding assistance as autocomplete toward viewing it as a semi-autonomous [coding agent](https://www.wikiprompt.org/wiki/ai-coding-assistant).

## Progress and variants

Early models solved only a small fraction of SWE-bench's original task set, but scores rose rapidly through 2024 and 2025 as labs specifically targeted agentic coding capability, aided by advances in [reasoning models](https://www.wikiprompt.org/wiki/reasoning-model) and longer effective [context windows](https://www.wikiprompt.org/wiki/context-window) that let models hold more of a codebase in view. This rapid progress led to the creation of harder variants, including SWE-bench Verified, a human-validated subset intended to remove ambiguous or unsolvable tasks, and SWE-bench Lite, a smaller and faster-to-run subset used for quicker iteration.

## Significance

SWE-bench is frequently cited in model release announcements from labs including [OpenAI](https://www.wikiprompt.org/wiki/openai) and [Anthropic](https://www.wikiprompt.org/wiki/anthropic) as a headline measure of coding and agentic capability, alongside broader [evaluation](https://www.wikiprompt.org/wiki/llm-evaluation) suites. Its grounding in real, verifiable software engineering work, rather than curated puzzle-style problems, has made it one of the more trusted proxies for practical usefulness in software development, though critics note it still covers primarily Python projects and issues resolvable within a bounded patch, and may not capture longer-horizon engineering judgment, architectural decisions, or work spanning many files and services beyond what the benchmark's task format allows.

## Impact on the field

SWE-bench's popularity contributed to a broader trend of evaluation moving toward realistic, task-grounded benchmarks over synthetic ones, mirroring similar developments in other domains as researchers sought to counter [benchmark](https://www.wikiprompt.org/wiki/benchmark) saturation and [contamination](https://www.wikiprompt.org/wiki/training-data) concerns affecting older static test suites.

---
Source: https://www.wikiprompt.org/wiki/swe-bench
License: CC BY-SA 4.0 (https://creativecommons.org/licenses/by-sa/4.0/)
Last updated: 2026-09-02T20:30:13.021953+00:00