SWE-bench, short for Software Engineering Benchmark, is a AI benchmark introduced by researchers at Princeton in 2023 that measures a language model's ability to resolve real software engineering tasks. Unlike synthetic coding benchmarks such as HumanEval, SWE-bench draws its tasks directly from actual GitHub issues and their corresponding merged pull requests across popular open-source Python repositories, asking a model to generate a code patch that resolves the issue and passes the project's existing test suite.
Design
Each SWE-bench task presents a model with a codebase at a specific commit, an issue description written by a real user or contributor, and the requirement to produce a patch; success is measured by whether the patch causes the relevant tests to pass without breaking existing ones. This makes SWE-bench considerably harder and more realistic than function-level benchmarks, since resolving an issue often requires understanding a large, unfamiliar codebase, identifying the relevant files among many, and making changes that integrate correctly with existing code, rather than writing an isolated function from a clear specification.
Relationship to agentic coding
Because real-world issue resolution typically requires multiple steps, exploring a codebase, running tests, and iterating on a fix, SWE-bench became closely associated with the rise of agentic coding systems rather than simple single-shot code generation. Tools such as GitHub Copilot's agent modes, Cursor, and Claude Code are commonly evaluated against SWE-bench or its variants as a proxy for real-world usefulness, and the benchmark's rise tracked the broader shift in the industry from viewing LLM coding assistance as autocomplete toward viewing it as a semi-autonomous coding agent.
Progress and variants
Early models solved only a small fraction of SWE-bench's original task set, but scores rose rapidly through 2024 and 2025 as labs specifically targeted agentic coding capability, aided by advances in reasoning models and longer effective context windows that let models hold more of a codebase in view. This rapid progress led to the creation of harder variants, including SWE-bench Verified, a human-validated subset intended to remove ambiguous or unsolvable tasks, and SWE-bench Lite, a smaller and faster-to-run subset used for quicker iteration.
Significance
SWE-bench is frequently cited in model release announcements from labs including OpenAI and Anthropic as a headline measure of coding and agentic capability, alongside broader evaluation suites. Its grounding in real, verifiable software engineering work, rather than curated puzzle-style problems, has made it one of the more trusted proxies for practical usefulness in software development, though critics note it still covers primarily Python projects and issues resolvable within a bounded patch, and may not capture longer-horizon engineering judgment, architectural decisions, or work spanning many files and services beyond what the benchmark's task format allows.
Impact on the field
SWE-bench's popularity contributed to a broader trend of evaluation moving toward realistic, task-grounded benchmarks over synthetic ones, mirroring similar developments in other domains as researchers sought to counter benchmark saturation and contamination concerns affecting older static test suites.