Why we no longer evaluate SWE-bench Verified

OpenAI announced it is ceasing evaluation of SWE-bench Verified, citing growing contamination and flawed test design that misrepresent progress in AI-assisted coding. According to OpenAI's blog post, the benchmark's test data increasingly leaks into model training sets, and its tests fail to capture genuine improvements in frontier coding capabilities. OpenAI recommends transitioning to SWE-bench Pro, which it claims addresses these issues. For developers and solopreneurs building AI workflows, this shift underscores the need to critically assess benchmarks when comparing coding agents. Relying on contaminated evaluations can inflate perceived performance, leading to misguided tool choices. Instead, builders should seek more robust, independently validated metrics to gauge real-world utility. The decision also highlights a broader challenge in AI evaluation: as models improve, benchmarks must evolve to stay relevant and trustworthy.