I think this can resolve NO now since humanity is now putting effort into aligning AI.

@MartinRandall We aren't putting effort into aligning all AIs. The market would resolve YES if some crazy e/acc people made a really big LLM and didn't put any effort into aligning it, but it spontaneously decided to be nice to humans anyways.

@EmienerYunoski I see, I'd write the title as "will A super-intelligent AI instantly become empathetic..." then.

WARNING: Not by me and it "expires Dec 31st" and "resolves according to [creator's] judgment". Seems like a trap, bad market, should maybe be closed by admin.

@EliezerYudkowsky It's obviously just a joke, don't think it makes sense to do anything about it. Unless you think he's genuinely impersonating you.

I think the answer here, contrary to what @EliezerYudkowsky thinks, is yes. But I'm not betting on this market because it's too ill-defined.

For example, it would obviously be possible to create a narrow AI designed to kill as many people as possible by deliberately limiting its training data. It wouldn't be generally intelligent, but would be successful at its effort. That's obviously something to be worried about.

But if you just spit all the data humans have ever created into the AI and see what comes out, then why wouldn't it come out like the average human? The average human doesn't want to die and doesn't want others to die.

@SteveSokolowski the average human doesn't love the average animal and the average human doesn't hate the average animal, but the average animal is made of atoms that the average human would like to arrange differently.

@MartinRandall This is true and it is exactly why the idea that eating meat will be seen as morally reprehensible in the future is absurd. It will not, ever.

But an AI will be able to talk to humans, and that will in fact make a big difference, the same way it in fact would make a difference (in the long run, at any rate) if we could talk to animals.

@SteveSokolowski What do you mean "spit all the data humans have ever created into the AI"? Do you mean like do unsupervised learning like transformers do on text, just including other modalities? If so, I don't think it likely this give you an "average human". currently LLMs are not like the average human at all. What you need to predict data generated by humans is not being like the average human, its the ability to simulate all humans and all the parts of the world we interact with. You should listen to @EliezerYudkowsky talk with Dwarkesh Patel. They talk about that there.

I think we're going to put in non-zero effort