People already run all sorts of LLMs connected to uncontained, agentic things. Like, sure, if you have some kind of isolation system adequate to contain it, then there’s no risk. But…people are running stuff without such an isolation system, all over, now.
Hell, South Africa infamously used an LLM to write the legal rules that they were proposing to control AI, then didn’t validate the text by human. If there’s one place you would not want to use unvalidated AI output, that’s it.
I tend to be somewhat skeptical of the practical risk, but in If Anyone Builds It, Everyone Dies, Yudkowsky makes the case that existing systems are already too dangerous, not because of what they thenselves can directly do, but because they are being applied in ways that could lead more-sophisticated systems to emerge.
If you’re an AI researcher, one of the most obvious and imnediately-relevant applications of AI is using it to improve existing AI.
Think of our current CPUs. No human could ever have designed them. They are far too complex, contain billions of transistors. But…we built computers and then promptly used them as tools to build more capable computers.
Yudkowsky’s point is that you have people doing that sort of thing with AI now, and doing so in a way that produces very unpredictable results.
Now, my guess is that that doesn’t constitute much of a probability of showing up by surprise in that it’s very likely that instead of a paperclip maximizer that obliterates humanity, you get something that fails spectacularly in a way that causes itself to stop functioning. Like, I’d bet that we’d see something more like a series of similar, spectacular failures before we get something that can function long enough and well enough to represent an existential threat.
But…I will concede that that’s more of a gut feeling. I don’t have real numbers. I can’t make a really convincing argument to myself that I am definitely right. And if I’m wrong, the cost is very high.