By which years will AI + scaffolding be shown to have a better log loss than the Metaculus community prediction on <= 1 year predictions?
Does not necessarily need to be evaluated on every Metaculus question, just a representative sample of Metaculus questions not selected to favor the AI with at least 20 unique predictors. Resolves to years where AI+scaffolding has been shown to have a better log loss anytime before the end of that year (including previous years).
I will not trade on this market.
1,000
1.00@MartinRandall If someone can use a model trained on text before a date and uses no external information after that date, I'd accept the predictions of that model for a year after the date.
@NoaNabeshima Oof, then we could reresolve 2024 to yes in 2030 after we have a better understanding of AI??
@MartinRandall No because it would need to be *shown* before EOY 2024. The <=1 year predictions don't need to be related to the year this ability is demonstrated except that they can't be too jerrymandered
@NoaNabeshima I see. So if in 2030 I train an AI based only on data from before 2024, and then I have it make predictions on Metaculus questions in 2024, again using data only from before 2024, and it beats the human predictors from 2024, that resolves 2030 to YES, only.
Seems similar to https://manifold.markets/dp/will-we-have-betterthanhumanaggrega except easier since only on 1 year time horizon? Why was 2024 lower here before I traded?
@EliLifland it's harder because it's performance on Metaculus only. Performance on Manifold is easier, eg acc outperforms me substantially.
@EliLifland Yes, the AI isn't allowed to see the community prediction. LMK if this makes anyone unhappy.
@SG the model could be private for the beginning of the forecasting period or the scaffolding could be complex