AI staff 'genuinely frightened' for humanity's future, ex-Anthropic researcher tells BBC

I’m not an AI techie and I’m not well educated on LLMs/AI in general. The issue I’m having is the dichotomy between how people talk about it. I’ve heard completely opposite ends of the spectrum. From AI is quickly becoming advanced enough to threaten human society, all the way to its a glorified dictionary/search engine that can’t really do much of anything. I know Lemmy has a steep anti-AI slant. But, can someone talk about this in a way I can wrap my head around why the divide seems so big?

www.bbc.com/news/articles/c1kx0gyje9wo

I’ve heard a few explanations for the “AI will kill us all” narrative.

1. AI techies are in a cult. They genuinely believe this stuff because they’re crazy.

2. It makes the AI sound powerful and impressive, which is ultimately good for the AI companies.

3. They’re starting to face competition from open-weight models, so they want to scare politicians into crafting regulations. They think that they will be able to influence the regulations to favor them and squeeze out their competition.

And, of course, there’s

4. It keeps people talking about AI. Because whatever you do, don’t quit talking about AI, because if you quit talking about it, the line might go down! And the line going down is very, very bad.

Okay cool, thanks!

1. If the techies believe it and they are the “experts” what makes them crazy? Wouldn’t their opinion be the most informed?

2. Yeah that part definitely makes sense

3. That definitely makes sense from a capitalist perspective. But, is the argument against regulations that they would be bad for the concentration of power it would give select companies? Or just that they aren’t necessary because the tool is of little use so it’s just wasted time/money.

4. That makes sense but trillions of dollars are being spent. Anything that amount of money is sinking into is gonna be talked about a lot. Especially thousands of jobs cut because of it.

I think there is something more interesting going on in the Jacob Coxon story because it’s [the] phrase “everyone at anthropic believes that AI is an existential risk”… Every organization has to instill some kind of beliefs among its members in order to do the work it does, right? Everybody at Goldman Sachs believes the free market is great, capitalism is the best system, and that they are the wealth creators who deserve big bonuses. Everyone in the Mormon church believes they should all tithe 10% of their income, and that they each get a special planet when they die. And everyone who worked at Philip Morris in the 80s believed that cigarettes tasted great and did not cause any of that nasty cancer. There’s Upton Sinclair quote, “It’s difficult to get a man to understand something when his salary depends on his not understanding it.” Well, the opposite is also true. It’s guaranteed that someone will believe something if their salary depends on them believing it.

Now, let’s just grant that all the people in these organizations definitely know things about finance and Mormonism and cigarettes and AI that you might not know because they are insiders and you are not. They have access to information and experiences that you do not have. And that might even make it a little bit difficult for you to win an argument with them. But arguing with their belief – trying to figure out whether or not its truth value is true or false – is not the only way to take a skeptical attitude towards it. We can also ask, what work is the belief doing for them. What does the thing that they believe allow the organization they’re a part of to do that it wouldn’t be able to do if the people in the organization didn’t believe that thing?

And this started to become clearer to me when I started listening to the actual claims that Jacob Coxin was making because a lot of them don’t really add up.

www.youtube.com/watch?v=pauBYSQ1K_c