Why was this resolved? It's still 2 days before they announce the winners and score. Was it announced somewhere else and I missed it?

53% was just achieved

https://www.kaggle.com/competitions/arc-prize-2024/leaderboard

49.5% was just achieved

https://www.kaggle.com/competitions/arc-prize-2024/leaderboard

49% was just achieved

https://www.kaggle.com/competitions/arc-prize-2024/leaderboard

48% was just achieved

I’m surprised “>50%” is only at 64% right now. Someone’s already managed to get 50% on ARC-AGI with GPT-4o using specific prompts, and it probably won’t be that hard to replicate.

The result on the leaderboard is 42%/43%

Yea because the admins haven’t managed to replicate it yet, but they’ll probably hear the news and replicate it soon

Sorry are you talking about https://www.lesswrong.com/posts/Rdwui3wHxCeKb7feK/getting-50-sota-on-arc-agi-with-gpt-4o or something else? The former is the first entry on arc AGI pub at https://arcprize.org/leaderboard

There are a bunch of limitations for the prize, one of which is that you can't use a closed source model. Now with LLama 3.1 405b the chance is probably higher, but then you also have limited compute for Kaggle that if the rules don't change don't allow afaik to run this model.

https://redwoodresearch.substack.com/p/getting-50-sota-on-arc-agi-with-gpt
ineligible for the prize though

neat benchmark,

IMO not that hard to solve, just have the synthetic data for it trained in just like ChessGPT or anything alike

They wouldn't count that.