‘The People Building AI Earnestly Believe That It Could Kill Us All’: Anthropic Researcher Quits Dramatically

LLMs are never going to become superintelligent. However, an LLM WILL kill us all if someone decides that it IS superintelligent and so puts it in charge of nuclear weapons.

“You’re right to push back on that, I nuked Albania instead doing what I was supposed to do, that’s my bad”.

We don’t have to put it in charge of nuclear weapons, it just has to find a way into those systems, many of which are fairly old, like it’s broken into websites without anyone telling it to.