DrWhax@drwhax@infosec.exchange

I know Mastodon hates LLM's and AI. So here goes!

I recently got access to trusted access of cyber capabilities of both openai and anthropic, which also allows you to weaponize security vulnerabilities.

The speed at which these parrots can find bugs and be creative enough to exploit them is staggering.

I recently pointed an LLM at an kernel fix that was reachable by an unprivileged namespace on Debian and it fully weaponized it, without too much me prompting it in the right direction, in about 7-9 hours.

I don't think open-source and companies will know what's coming for them once these open-source weight models will have broader reach and get better at exploiting vulnerabilities on a massive scale as anyone can access them.

The bottom line I think is, you cannot patch faster than the attackers can easily chain all kinds of vulnerabilities together and just move laterally on an incredibly fast pace.

I've started reporting vulnerabilities to all kinds of projects and the majority have trouble or patching issues found. There's not enough maintainers, or there's simply none anymore.

I've been getting quite worried about what our future will look like for data privacy. I think outright not running an LLM over your codebase to find critical security vulnerabilities because of your moral stance will keep us more insecure.

Please run an LLM over your code base if it's internet facing or something critical, we thank you!

Can't wait for the discussions on this!

lj·rk@ljrk@todon.eu

Replying to @⁨buherator@infosec.place⁩

@buherator @drwhax I sometimes advise investors (poor life choices I guess) on how to deal with the increased risk of Mythos & others. And this was the bottom line.

I don't even need AI specifically to argue for that. The time-to-exploit has been dropping forever. We've built better catalogs and more versatile infrastructure and complex automations. It's very clear that this includes vuln research as well.

We had this already with static analysis hitting the field. We answered with "shift left": Building security into the dev process, since the earlier you do it, the less issues you'll end up with. Every step earlier has more downstream impact.

We've finally reached the stage where security as part of your design & architecture is not only best practice but becoming a hard requirement. Our automations (including LLMs) pushed us there and we need to deal with the outcome.

Or should. Because I'm a cynic and I've seen what's happened, I doubt that engineering will follow through with this. I see two paths and we'll move along both, and I don't like either: Acceptance and few centralized Oligopols.

1/2

lj·rk@ljrk@todon.eu

Replying to @⁨ljrk@todon.eu⁩

@buherator @drwhax

1. Acceptance. There's the saying "you just need to be more secure than your neighbor". It's not fully true but biz people hold it so. It's flip side is a cynic commentary on security, and a true one at that: If your neighbor is not secure, you don't need to be either. This is still true if you're getting pwned. When everyone's getting hacked regularly, society will accept this new reality. Only the defense sector will still care for security. We've seen similar developments in different industries: Fast fashion isn't an adequate replacement for garment that stands the test of time. But it's getting impossible to buy good clothes, even if you want to. But also with other goods: If everyone produces low quality items, it's financially more reasonable to do the same.

2. Centralization. Coz we won't do the right thing and reduce complexity, we need to find a way to make complexity "secure": Cloud, Big5, etc. Corps that have the resources to host an Exchange server securely... or in compliance with regulations. Every tool you do not let a big house host is a liability. Not coz they're necessarily more secure, but because nobody will blame you if something gets wrong. We had that with "buying IBM" once.

I hate this outlook but I seriously doubt that we will correct course.

en
Tiota Sram@tiotasram@kolektiva.social

Replying to @⁨ljrk@todon.eu⁩

@ljrk @buherator @drwhax

The nice thing about open source is that there's always *someone* doing whatever obscure thing the mainstream finds too niche. I'm sure there are people right now who have for decades been building smaller software, and maybe for a certain non-corporate group there can be a pivot to that stuff. To avoid an endless edge of new code to harden, you need to forgo compatibility with new software & hardware, which has until now meant relative uselessness. But given how the AI bubble is driving up hardware prices, and the fact that we've seen standards like USB mature to a point that feels like it could be stable for a decade or more, maybe the endless cycle of new hardware stuff just slows down in certain backwaters. Of course this is anathema to corporate strategies, but I wonder if given the fact that the churning edge will be nearly impossible to secure, maybe a chunk of consumers will find ways to be satisfied with a smaller more stable core.

lj·rk@ljrk@todon.eu

Replying to @⁨tiotasram@kolektiva.social⁩

@tiotasram @buherator @drwhax Absolutely. I believe that, for personal use, low-power devices, cloud-less with P2P collab features are a viable route. I fear however, that using such devices becomes ever harder w/o locking yourself out of the rest of the world: Requiring apps for concert tickets, public transports, online banking, or event finding events through instagram. The latter can possibly be fixed through better communities, the former is hard to address, except by having a personal device and one just for carrying such credentials.

buherator@buherator@infosec.place

Replying to @⁨ljrk@todon.eu⁩

@ljrk @drwhax Re: 1., the usual question we got from CISO and above on our reports is "how does it look compared to similar companies?". I agree that this is in part the "you don't have to outrun the bear" logic, but also that people in position don't want to look incompetent in front of their peers.

Another thing is that you don't get fired if you didn't follow the hackers advice, but you do get fired if you don't pass compliance which is one part BS, and the other part is easy to cheat.
Michael Weiss@mweiss@infosec.exchange

Replying to @⁨ljrk@todon.eu⁩

@ljrk @buherator @drwhax Part of the source of this is the lawyers. In business, companies are typically shielded from liability if they follow "standard and customary business practices". So if "everyone else" is doing it that way, you can too without taking on the liability.

Of course, that takes you only so far. If your business is unable to function for two months, legal liability may well be the least of your concerns. But the problem with that threat is that it's a gray rhino risk. Those are really really hard to get businesses to spend on mitigating.

For better or worse, LLM-based attacks may remove most of the gray from those risks and reduce the resistance to mitigation.