Why it matters

SWE-bench is de facto SWE agent benchmark. Understanding shapes evaluation.

Advertisement

The architecture

2294 issues (300 in Verified).

Docker environments.

Test-based scoring.

SWE-bench flowIssue + repotask inputAgent produces patchdiffRun testspass = solvedVerified subset is human-validated; use for reliable comparisons
SWE-bench.
Advertisement

How it works end to end

Docker sandboxes.

Original tests as scorer.

SWE-bench Multimodal, Java also emerging.

Leaderboard active.