This question will resolve as the state-of-the-art average score on the GAIA benchmark (on the test set, not validation set) by an AI system, including any post-training enhancements but excluding any human assistance. This will be based on credible publicly available results prior to January 1st 2025. The primary credible source will be the official leaderboard, but other sources, including but not limited to arXiv preprints and papers may also be considered.

Background Information:

See GAIA,

GAIA is a benchmark which aims at evaluating next-generation LLMs (LLMs with augmented capabilities due to added tooling, efficient prompting, access to search, etc). (See our paper for more details.) GAIA is made of more than 450 non-trivial question with an unambiguous answer, requiring different levels of tooling and autonomy to solve. It is therefore divided in 3 levels, where level 1 should be breakable by very good LLMs, and level 3 indicate a strong jump in model capabilities. Each level is divided into a fully public dev set for validation, and a test set with private answers and metadata.

Best score on March 15th 2024 is GPT-4-turbo based and achieved 32.33%.

Part of the AI Benchmarks series by the AI Safety Student Team at Harvard on evaluations of AI models against technical benchmarks. Full list of questions:

• https://manifold.markets/JonasVollmer/what-will-be-the-best-score-on-the

• https://manifold.markets/JonasVollmer/how-many-metr-tasks-will-be-complet

• https://manifold.markets/JonasVollmer/what-will-be-the-best-score-on-the-d38814e2aff2

• https://manifold.markets/JonasVollmer/what-will-be-the-best-score-on-the-dc351f43cd0e

• https://manifold.markets/JonasVollmer/what-will-be-the-best-score-on-the-8f2bf7f44d8e

• https://manifold.markets/JonasVollmer/what-will-be-the-best-score-on-the-a21d0872429b