We will evaluate by:

1) selecting an article from news

2) paste the article into an LLM along with the question: "based on this article, how confident are you that this event has happened from 0-100%? Please answer by filling in this template: {" text of disaster from list above", probability: X%}

If the average probability across leadingLLMs or the best AI agents at the time, across five tries each, is over 70%, this can resolve true.

So overall we use the LLM to evaluate the claim based on an article. I'm going to try to seek truth as much as possible and not accept a trick or false or unintuitive resolution.

Here is the list of disasters:

1. A massive cyberattack shuts down the power grid across multiple states

2. A global pandemic rapidly spreads in the US with a high fatality rate

3. A mega earthquake devastates the West coast along the San Andreas fault

4. The Yellowstone supervolcano erupts and spews ash across the country

5. A large asteroid impacts the Atlantic ocean, causing tsunamis along the East coast

6. A terrorist organization detonates nuclear weapons in multiple cities

7. Category 5 hurricanes make direct hits on New York City and Miami

8. Runaway wildfires burn across multiple Western states

9. A coup d'état ousts the president and suspends the Constitution

10. The New Madrid fault ruptures causing massive earthquakes in the Midwest

11. A civil war breaks out between liberal and conservative states

12. Foreign troops invade the West or East coast of the US

13. The US dollar collapses triggering an economic depression

14. A bioterror attack spreads deadly engineered pathogens nationwide

15. Massive riots and unrest erupts in cities across the country

16. A killer swarm of locusts destroys crops in the Great Plains

17. A large meteor strikes a major city causing catastrophic damage

18. The Cascadia Subduction Zone ruptures, unleashing a massive tsunami

19. A militarized faction stages a nationwide coup and seizes power

20. The Yellowstone volcano erupts, sending the US into a volcanic winter

21. A new civil war erupts between red and blue states

22. A massive flood inundates much of the South after levees fail

23. Martial law is declared after a disputed presidential election

24. A deadly heat wave kills thousands across the Southwest

25. A reactor meltdown contaminates areas around nuclear power plants

26. A second civil war erupts this time along ideological lines

27. Coastal cities are abandoned after being swamped by rising sea levels

28. Killer robots and drones turn against humans in a technological singularity event

29. A terrorist group detonates a dirty bomb in a major financial center

30. A global food and water crisis leads to nationwide starvation and unrest