This question will resolve as the state-of-the-art accuracy on the SWE-Bench unassisted benchmark by an AI system, including any post-training enhancements but excluding any human assistance. This will be based on credible publicly available results prior to January 1st 2025. The primary credible source will be the official leaderboard, but other sources, including but not limited to arXiv preprints and papers, may also be considered.

Background information:

See SWE-bench.

SWE-bench is a dataset that tests systems' ability to solve GitHub issues automatically. The dataset collects 2,294 Issue-Pull Request pairs from 12 popular Python repositories. Evaluation is performed by unit test verification using post-PR behavior as the reference solution. Read more about SWE-bench in our paper!

Best reported system on March 15th 2024 is Devin achieving 13.86%. The official best on the official leaderboard is Claude 2 + BM25 Retrieval with 1.96%.

Part of the AI Benchmarks series by the AI Safety Student Team at Harvard on evaluations of AI models against technical benchmarks. Full list of questions:

• https://manifold.markets/JonasVollmer/what-will-be-the-best-score-on-the

• https://manifold.markets/JonasVollmer/how-many-metr-tasks-will-be-complet

• https://manifold.markets/JonasVollmer/what-will-be-the-best-score-on-the-d38814e2aff2

• https://manifold.markets/JonasVollmer/what-will-be-the-best-score-on-the-dc351f43cd0e

• https://manifold.markets/JonasVollmer/what-will-be-the-best-score-on-the-8f2bf7f44d8e

• https://manifold.markets/JonasVollmer/what-will-be-the-best-score-on-the-a21d0872429b