SWE-Bench Pro V2: A Cleaner, Harder-to-Game Coding Benchmark

[arXiv] Soham Dan, Jeff Da, Daniel Yue Zhang and other authors have published a paper on arXiv, releasing V2, a revised public version of the coding agent benchmark SWE-Bench Pro. The authors state that as agent capabilities improve, they encounter task issues that earlier agents never touched, and interact with the environment in ways the original evaluation framework did not anticipate. Through agent auditing, expert adjudication, and blind-test reimplementation, the team fixed or removed 480 of 731 tasks; 23 contract engineers invested 404 hours for this, and each revised instruction was verified by another engineer solving it from the text alone. The new protocol restricts network access to the model endpoint and re-scores each patch in a clean container, preventing agents from retrieving upstream fixes or scoring points from changes outside the patch. The paper states that on the 642 V2 tasks, frontier agents achieve a maximum resolution rate of 99.4%, and on the Hard subset of 51 tasks, system scores range from 52.9% to 98.0%.