AI staff 'genuinely frightened' for humanity's future, ex-Anthropic researcher tells BBC

I’m not an AI techie and I’m not well educated on LLMs/AI in general. The issue I’m having is the dichotomy between how people talk about it. I’ve heard completely opposite ends of the spectrum. From AI is quickly becoming advanced enough to threaten human society, all the way to its a glorified dictionary/search engine that can’t really do much of anything. I know Lemmy has a steep anti-AI slant. But, can someone talk about this in a way I can wrap my head around why the divide seems so big?

www.bbc.com/news/articles/c1kx0gyje9wo

I’m not pro-AI by any stretch for a huge number of reasons, but in my day job as a technical product manager I have to use a number of AI-powered tools and workflows. Whereas 18 months ago they were a novelty that the bosses were enamored of, today they get real work done and speed up a number of key areas, mainly like fancy automation workflows (which have been around forever). The rate of improvement and capability for certain use cases has been very, very rapid. We do seem to be reaching a plateau though. The tools and models from 6 months ago are just about as good as the ones from today. And China’s models from today are better than the US’s models from 6 months ago, but at a fraction of the cost.

So when I hear people talking about the “end of the world”, I think a realistic case for that is that the AI tools that come out in the next few years will be good enough to do a fair amount of white-collar work, and we won’t have a plan in place to deal with all of the now out-of-work people that leads to. Whole classes of jobs in areas like data entry, telemarketing, and customer support will be almost entirely automated out of existence. New classes of jobs will be created, of course, but it’s hard to imagine them existing in sufficient numbers to make up for the losses. And this is just for LLM-workloads.

In other areas, other types of AI are already enabling robotic automation of factories and in the near-ish future might make self-driving truck fleets practical. This would be devastating because factory workers and truck drivers make up like 30% of the US workforce – and again, there are no signs of new classes of jobs that could absorb the out-of-work workers, and no plans for a social safety net or wealth distribution.

So I’m not worried about a rogue AI unleashing some bioweapon or launching a nuclear attack. I’m worried about a bunch of CEOs deploying barely-acceptable AI to displace millions of workers as soon as it is safe(ish) to do so.

Are we reaching a plateau though? I would prefer it, and it looks like we could, but the progress from GPT-5 to 6 or Opus to Fable has still been very noticeable on more complex tasks. They are just less public due to stricter limits or straight up pay by use. And where does that plateau lie? Within our grasp or maybe just outside of it?

What has me sharing the concerns at least in part are the facts that

1. those employees have access to less restricted models and much more resources than the average joe. That huggingface attack was committed by a huge swarm overcoming the guardrails and engaging in cheating and crime to reach their goal.

2. they are working on cutting the human out of the loop for AI that improves itself. Maybe that won’t work entirely, but what if it does?

3. the warnings are starting to come not from Altman, Musk et al, but from people who could have something to lose for blowing the whistle. „Could“, because I don’t know them either and there could still be ulterior motives, not just as obvious as with the psychos at the top.

The whole thing comes down to one set of questions, in my opinion: what actually IS intelligence, consciousness, sentience and where do they come from? Since we have not been able to properly define that, it would be foolish to think that we have a definite answer to the AI questions.

We like to consider ourselves something extraordinary out of the animal kingdom. And while I prefer the existence as a human to being a dog or mollusk, I have always considered that assumption quite arrogant.

The truth is: we don’t really know. And that means we can not know if something comparable or stronger could be created or emerge from something other than squishy meat.

Those are quite philosophical questions. Instead of thinking first and acting after that we are much too busy building datacenters to conjure even more boobie snaps out of thin air and somehow weaponize that newfangled thing.

THAT is what scares me.

In worst case the humankind will find out consciousness isn’t too complex and way easier to achieve.

And there are different levels of consciousness as I learned working with impacted people.