@Bayesian Just to verify, do you consider the Scale.ai leaderboard to be up to date? Other than the missing Grok 4 score, that's what I am assuming. Are you counting the unconfirmed result of Deepthink of 34.8% no tools that was posted by Google? Also, are you counting the tool use variants?

@Jolliest hmmmmm it is curious that grok 4 is missing from the leaderboard. If it wasn't for that i'd be sure it's up to date but I have to reserve judgement right now bc i can't really tell. but more recent models are present so idk i'll preliminarily say it's up to date. the intent is definitely to prioritize only considering scaffolds + ai systems allowed under the scale ai leaderboard section.

Are you counting the unconfirmed result of Deepthink of 34.8% no tools that was posted by Google?

I am not counting this unless it goes on the leaderboard, at this time

0-10% can probably be resolved no

@redcathode Unfortunately we can’t resolve an option early in the case of multichoicr markets

@Bayesian ah, thanks anyway

Currently this market has an expected average score of 53.6 which I think is quite high. Especially given how the neural scaling laws seem to be coming home to roost.

lots of arb possible with my market https://manifold.markets/jim/when-will-humanitys-last-exam-be-sa

Are models using tools and/or performing web search eligible under current resolution criteria?

@Metastable I think no matter what tools you add to AI, it still remains AI

@mathvc ability to livechat with human experts?

@jim with an exception of using humans 🙂, then it’s definitely not artificial intelligence

@mathvc live access to web is maybe somewhere on that spectrum tho

@jim why you call it live access? It doesn’t go to math forum and make a post about math problems.

You can replicate internet access by scraping it and using as a giant database

@mathvc yeah but using giant database or the web seems like it's less reliant on the AI model's innate knowledge and intelligence, more reliant on human knowledge and intelligence.

i've edited the market description a bit to not be dependent on my own discretion for what model counts or doesn't count. now it uses a consensus of reliable sources or moderator consensus, instead of my own opinion. 🤷‍♂️ probably won't come up anyway but i realized i was amassing a decent position so

Anything 30+ is honestly scary territory. FrontierMath is really impressive and all, and no doubt surprising, but kinda an oh that happened type thing. This is the kind of test that would make me seriously reconsider my beliefs about AGI, great market!

ooo maybe a market on which of those two is solved at 80% or above first

@Bayesian sounds like an incentive to finetune my deepseek-giga-overfitter-hle-memorized-v1 model by EOY

@Ziddletwix yeah but that would be CHEATING! and the leaderboard thing would CATCH IT

@Bayesian most likely, but maybe they'll just put an asterisk and scold it in a footnote for being sus & bad. unclear how enforcement is actually handled in practice

fkkkk they might put the footnote saying it's sus affffff then what are we gonna do

@copiumarc I don’t think HLE is harder than Frontier Math.

/Bayesian/which-of-frontiermath-and-humanitys

@mathvc @copiumarc may the person with the best model of reality win

Surely o3 will get >20%?

@qumeric if the benchmark is knowledge heavy it might not do that much better than 4o? prolly will tho. just some low chance that it doesn't