The reason I can say with 100% confidence that coding is not solved ans I am not actually just a doomer getting left behind is because I actually look at the output, and if the code gods were actually writing code that was so good we can't comprehend it, then I would not be averaging two or three 0-click account or full-fleet takeover vectors a day as a total scrub.

On the other side, i assume they are all also doing full auto LLM pentesting, so that also clearly doesn't work. It might be able to brute force its way into some esoteric memory allocation bug with access to the source code, but like how 99% of security is just telling people not to write their passwords on a sticky note on the monitor not high-tech exploits, most of the bugs I am finding are things that an LLM would never imagine are an attack vector because they are so silly and shouldn't be possible. When I have sent the meta models, Claude, and open weight models to go bughunting, even directing them straight towards something i just confirmed to see if they find it,, they get waylaid by ambient strings that say how things should work, or fall back to trained patterns of how things in the shape of what they are looking at normally work. They often don't even try doing things like "just call super-secure-secret-vault" because they assume they can't. The LLMs are good for stuff like first pass RE, but can't really chain together an attack when part of the chain involves "imagine this thing is shittier than the shittiest possible thing you could imagine and you can just press the god mode button"

@jonny Of course they can't do that, they can't imagine anything. You're anthropomorphising them! Too much time around them even as a hater gives you the AI psychosis and you start thinking about them like a being that can make assumptions.

@tael
Please interpret my use of "imagine" as "produce the corresponding text" and I am using language as a system of metaphor and shorthand rather than affirming the literal meaning of every word uncritically.

@jonny I'm just saying, it infects the way you speak about them. Language has a powerful sway over our brains; the ELIZA effect is a gradient, not just a binary between smart correct person and credulous rube. I'm not saying it to insult you! I think it'd happen to me too if I exposed myself to the fucking Elephant's Foot like you do to bring us this information. You didn't do anything wrong if you get irradiated. It just had to be done to protect people.

@jonny Government: we need to secure our systems against #LLM #cyberattacks... 🤔 so we'll use LLMs to find and fix the vulnerabilities.

Jonny: just use LLM generated code, because it is so bad LLMs won't be able to hack it. And human hackers will be back in a job, bingo, return to status quo.

This could oscillate!

@jonny That's the real "red team"; Convincing your boss that AI can do everything, and do it better than you. They adopt it, and then riddle everything with bugs

@jonny by "zero click" you mean you just say to your phone "chatgpt please hack this for me" right?

@nev
As in you go to log in to your muse account in the morning and you cant, after you did nothing.

@jonny Only the true believers think it's so good we can't comprehend it. The cynical boosters think that it's good enough that capitalism will accept it in place of having to pay developers, a standard it might conceivably achieve because capitalists care about profit, not quality. They don't understand that it's actually so much MORE expensive than paying developers that it's not even viable just to fuck labor.

@jonny
@Viss @cR0w

@theorangetheme @jonny @Viss Shit, I can't even get an interview for the same kind of work I'm already doing. I'm not the one to tag to help someone make a career change, unfortunately.

@cR0w @jonny @Viss Apologies. I wasn't entirely sure whomst to tag.

@theorangetheme @jonny @Viss No worries, just frustrated by the state of things. 🍻

@cR0w @jonny @Viss *hug*

@theorangetheme @cR0w @jonny i have a bunch of advice but im not sure it matters anymore. this year i got told one potential customer wasnt going to sign because they were going to use ai as their vciso, and then another potential customer asked me to come up with some language defending using a human for security at all in preparation for that guy expecting the ceo to ask

@Viss
@theorangetheme @cR0w
I can't imagine how exasperating it must be to be like "but look at it, the thing is swiss cheese even with the latest hotness in AI security evaluation, but I as a human can see it right there." and still having people not see value in the profession at all.

@jonny @theorangetheme @cR0w i just have to calmly explain:
- 100% of every org that has ever had a massive deletion event or had an ai go off the rails experienced that because they gave the ai too much access
- if you dont have a security expert there to interpret the results, you will have no way of knowing if what the thing just told you is an outright lie or not
- the risks are quite extreme when dealing with security, and accounting for these existing problems llms have

@jonny how do you feel working for the Israeli Cyber-Arms industry?

@AuntyRed
Not in favor

@jonny tbf when you have 1 red teamer per thousand devs (and now a crèche full of agents) slapping out product at million miles an hour. Most of it’s going through to the keeper.

@jonny also hating on your product I find makes you a target from management as being a troublemaker

@Jplonie @jonny

All of which is a sign of both a failing corporation and general insanity of investors.

@jonny I dunno man, listen to "Shake Your Foundations" by AC/DC a lot and do a bag of coke? "Red Team Shoe Diaries" doesn't have quite the same ring to it and you might wake up David Duchovny (he could pull it off as Denise Bryson in 1990 but not now)

@jonny I think that's called a black hat. Have you considered selling exploits or running a botnet?

(Obviously this is a joke, that would be very unethical. Everyone knows the ethical thing to do is being paid to protect Meta's software stack so they can destroy the remnants of democracy faster.)

@jonny In all seriousness, actual grown up SAST & DAST testing like Coverity and so on is pretty unglamorous. I actually had an item in my graph algorithms notes to look at Jorgensen et al.'s "Software Testing: A Craftsman's Approach" precisely to unpack all the complexity therein

@jonny as far as I can tell: you start a security company that pretends to be more than just you, put a lot of manager-friendly verbiage on its website, and start cold-calling companies until someone says yes

@joepie91 @jonny Doing some presentations at Black Hat would help too ...

@sporfk @jonny oh, yeah, almost forgot about that one!

@jonny

It would be wise to consider Mephistopheles when at such a crossroad. After all, cojote won the poker game and escaped hell.

@jonny I think you just figure out what the digital equivalent of a fluorescent vest and clipboard is and get to work. Forgiveness vs permission and all that!

@jonny "Sithteaming": Like redteaming, but fueled by hate, fear and anger.