On the other side, i assume they are all also doing full auto LLM pentesting, so that also clearly doesn't work. It might be able to brute force its way into some esoteric memory allocation bug with access to the source code, but like how 99% of security is just telling people not to write their passwords on a sticky note on the monitor not high-tech exploits, most of the bugs I am finding are things that an LLM would never imagine are an attack vector because they are so silly and shouldn't be possible. When I have sent the meta models, Claude, and open weight models to go bughunting, even directing them straight towards something i just confirmed to see if they find it,, they get waylaid by ambient strings that say how things should work, or fall back to trained patterns of how things in the shape of what they are looking at normally work. They often don't even try doing things like "just call super-secure-secret-vault" because they assume they can't. The LLMs are good for stuff like first pass RE, but can't really chain together an attack when part of the chain involves "imagine this thing is shittier than the shittiest possible thing you could imagine and you can just press the god mode button"

@jonny Of course they can't do that, they can't imagine anything. You're anthropomorphising them! Too much time around them even as a hater gives you the AI psychosis and you start thinking about them like a being that can make assumptions.

@tael
Please interpret my use of "imagine" as "produce the corresponding text" and I am using language as a system of metaphor and shorthand rather than affirming the literal meaning of every word uncritically.

@jonny I'm just saying, it infects the way you speak about them. Language has a powerful sway over our brains; the ELIZA effect is a gradient, not just a binary between smart correct person and credulous rube. I'm not saying it to insult you! I think it'd happen to me too if I exposed myself to the fucking Elephant's Foot like you do to bring us this information. You didn't do anything wrong if you get irradiated. It just had to be done to protect people.

@jonny Government: we need to secure our systems against #LLM #cyberattacks... 🤔 so we'll use LLMs to find and fix the vulnerabilities.

Jonny: just use LLM generated code, because it is so bad LLMs won't be able to hack it. And human hackers will be back in a job, bingo, return to status quo.

This could oscillate!

@jonny That's the real "red team"; Convincing your boss that AI can do everything, and do it better than you. They adopt it, and then riddle everything with bugs

@jonny by "zero click" you mean you just say to your phone "chatgpt please hack this for me" right?

@nev
As in you go to log in to your muse account in the morning and you cant, after you did nothing.

@jonny Only the true believers think it's so good we can't comprehend it. The cynical boosters think that it's good enough that capitalism will accept it in place of having to pay developers, a standard it might conceivably achieve because capitalists care about profit, not quality. They don't understand that it's actually so much MORE expensive than paying developers that it's not even viable just to fuck labor.