@elle @GrapheneOS

"for an OS project that refused porting to non-PIxel h/w for so long b/c other platforms didn't meet security reqs, this pro-LLM stance is ludicrous."

Read or reread the thread; it might prevent you from responding with inaccuracies, but I think you did it on purpose.

@Xtreix ? care to elaborate?

before you make accusations about me intentionally misinterpreting things

@GrapheneOS

@Xtreix

"It also definitely isn't useful to do it with a free or cheap model available without either paying for it or running it on very expensive hardware"

sounds like pretty explicit, "if you're not using proprietary AI models, don't bother" AKA PR propaganda for those firms from a big-name project in the FOSS space. one that up-to-now, has a shining record for security + privacy.

@GrapheneOS have you really explored all alternatives to these models for surfacing security issues?

@elle @GrapheneOS

"sounds like pretty explicit, "if you're not using proprietary AI models, don't bother" AKA PR propaganda for those firms from a big-name project in the FOSS space"

No, that's not the case. GrapheneOS hasn't suddenly become pro-LLM and isn't promoting big AI companies, those are the false claims you're making.

@Xtreix

"Frontier models are very good at finding bugs but it requires an experienced developer to babysit it and extract the useful portion of the output."

"The volume of real vulnerabilities being found this way ... is quite scary"

Sounds pretty pro-LLM to me. "oooo big LLM so scawy, definitely couldn't find the same bugs with equivalent compute using non-LLM tools"

the majority of bugs haven't been triaged, ones that have are shallow(-ish). this is part of the PR hypecycle

@GrapheneOS

@elle @Xtreix @GrapheneOS

Relaying the reality of the situation that AI review has helped find vulnerabilities that evaded human examination does not somehow make GrapheneOS pro-AI. They dont like it. I dont like it. You dont like it. But the alternative is either lying or doing nothing. Would you rather them lie to us? What they said is objectively true. It sounds like you want them to conceal this from us and/or leave vulnerabilities unpatched.

@HybridStaticAnimate

> It sounds like you want them to conceal this from us and/or leave vulnerabilities unpatched

what a strange interpretation of "bugs could be found w/ non-LLM tools using equivalent compute".

also, of course they should fix any legit bugs. does that mean they need to come out defending big-tech LLMs as "the only tools that could have found these bugs" (effectively doing PR for those companies)?

does it mean they need to use the LLMs for code gen?

@Xtreix@infosec.exchange @GrapheneOS

@elle @GrapheneOS

"Can" is irrelevant. Any bug can be found with sufficient time, but attackers are using tools that find them faster, and find vulns that humans missed. Fixating on the "can" part ignores the nuance, like how *quickly* and how *many* vulns are being discovered with these tools. If this can be done without AI, why was it not done already? Why has attacker and defender discovery of exploits accelerated exponentially? There is no substance behind the claim this was achievable with other tools.

The objective reality of how these tools work and how effective they are is not a "defense" of LLMs and GrapheneOS is not doing PR for them. For them to say something different would be a lie and to not say anything at all would be concealing information.

GrapheneOS has already said any code made with AI still needs to be understood fully by the person submitting it, and it needs to meet code quality requirements. An LLM alone is not capable of meeting these standards and a developer needs to know what the code is doing, how, and why.

@HybridStaticAnimate

> If this can be done without AI, why was it not done already?

b/c there wasn't a multi-billion dollar, VC-funded campaign to subsidize the compute required to use other techniques?

what's the closest equivalent, OSS-fuzz from Google? also, open models, even frontier-grade, exist if you insist on using LLMs. why aren't those being utilized? why choose the ones specifically attempting to garner good PR by getting FOSS projects to use their shit?

@GrapheneOS

@elle @GrapheneOS

There arent OSS equivalents. They are months behind. They are still useful, yes, but the ones they are using are currently more effective. This isnt some kind of ad, they picked the most effective tool for the job and they will pick a different one if another becomes more effective.

@HybridStaticAnimate @GrapheneOS hmm let me check:

- being "months" behind still doing useful bug finding
- being "cutting-edge" and supporting fascists actively destroying the world for oligarchical benefit

which to choose, which to choose...

@elle @GrapheneOS

We, "in the business", call this a false dichotomy.

@HybridStaticAnimate

no, those are the true "dichotomous" solutions that are in the same arena.

frontier-grade, open-weight LLMs that are some relatively small degree behind the proprietary models created by fascist war criminals.

you can obfuscate, and run defense for this all you want: that is the choice you or anyone else is making.

there are legit situations where there is no alternative. this ain't one. there is a viable alternative that is only slightly sub-optimal.

@GrapheneOS

@elle @GrapheneOS You are making false claims.

@Xtreix

get blocked ducky

@GrapheneOS

@GrapheneOS to clarify the other methods I'm referring to are symbolic analysis, fuzzing, and static analysis tools.

I think this post from @david_chisnall puts it very well: https://infosec.exchange/@david_chisnall/117183568828323924

@elle @GrapheneOS what's the relation of Google being Google and a language model?

@dasmatus

are you serious? what is the relation b/w GrapheneOS taking security so seriously that they wouldn't port their OS to h/w that didn't have Pixel's special sauce, and now being fine using LLM code + bug reports + dedicate human time to "babysit" the LLMs?

my intention is not to "bully" @GrapheneOS. we need folks like you in FOSS building rock-solid projects. I'm merely voicing criticism, and hoping the team explores other alternatives.

leave big (proprietary) LLMs in the bin

@elle @dasmatus GrapheneOS is based on Linux, AOSP, Chromium and hundreds of other upstream projects heavily using AI generated code.

Our OS development team hasn't been using any AI code generation but yet the OS is heavily using it because the vast majority of the code comes from upstream projects. Linux and Chromium are both massive projects.

Firefox is heavily using it too, so that isn't relevant to this. It's nowhere close to what we would need to use it as a replacement regardless.

@GrapheneOS

I know, that's why we need as many folks on the side of "we're not using this shit, those big-tech LLM fucks can get bent"

it's also clear FOSS teams don't have the resources to rebuild / fork foundational projects like Linux / Chromium / Firefox, each one containing millions of LoC.

@dasmatus

@GrapheneOS this is also why I'm looking for alternatives to those projects, especially in anything I care about.

again, I'm not condemning you for being caught up in the same bullshit as the rest of us. there is no perfect or easy answer to any of it.

@dasmatus

@elle @GrapheneOS @dasmatus if you have watched reported vulnerabilities over the last few months, you'll see that there is a huge increase in the last few months due to LLMs. LLMs finds vulnerabilities that older methods such as static analysis and fuzzing didn't and don't find.

Since security is paramount to GrapheneOS and adversaries are using LLMs, the only way to defend is to use LLMs for auditing too.

LLMs are here to stay, let's boost open models to own the means of production.

@elle @dasmatus Our policy is only accepting high quality code. We've always had to do careful code review and avoid trusting submissions.

We aren't happy with the flood of slop which gets submitted to us via our issue tracker, security@ email and pull requests. However, a lot is reporting or fixing valid issues.

For pull requests fixing valid issues, we can either get them to improve it to meet our standards or do it ourselves. We often start with rounds of review but finish it up ourselves.

@elle @dasmatus In the past couple months, there has been a massive flood of valid vulnerabilities being discovered by frontier AI models. Nearly all of those are from human written code, often from many years ago.

Linux has heavily shifted away from adding new features and improving performance to fixing the bugs found by these models. They're being forced away from sticking with C and a monolithic kernel design. It's on ongoing crisis and people can regularly grab working exploits off GitHub.

@elle @dasmatus Cellebrite and other exploit development companies are having growing success exploiting all kinds of targets. They're openly talking about how they're using AI tools for reverse engineering, vulnerability discovery and exploit development.

It's a very large and rapidly growing short term advantage for attackers. The consensus among security researchers is that it's short term because vulnerabilities are being rapidly found by defenders too including before merging or release.

@elle @dasmatus We have a responsibility to protect our users. We're aware frontier AI models can be used to find many vulnerabilities. We made several accounts on cheap plans with a fixed amount of usage, applied for and received cybersecurity access and began testing it. We've found what others have which is that these tools find many actual vulnerabilities including subtle domain specific issues. It's largely beyond the scope of what traditional static analysis tools even attempt to do.

@GrapheneOS @dasmatus I understand, what I'm suggesting is that we as a community explore other solutions, possibly involving open models (hopefully not), that can also provide serviceable results for vuln disco.

for example, taking a symbolic + static analysis, and fuzzing approach that shells out to an open model LLM (or dev a fine-tuned small model) for path exploration when the other tools get stuck.

in other words, use/develop purpose built tools instead of rigging a GPT

@elle @dasmatus As with anyone else, we make mistakes and there are many things we don't know about. There are undocumented behaviors of library and Linux kernel APIs. We have the wrong mental model about how some things work. We can review a pull request a dozen times with great care and still miss things quickly discovered with one these tools. We confirm it's a real issue and figure out how to fix it ourselves. Our fix can also be checked. We write small volumes of code so it's very cheap.

@elle @dasmatus Our OS development team and most of our other developers aren't using AI coding assistants in the form of an IDE. They'd need to be run in a VM as a secondary workspace with a separate GitHub account or a similar approach.

We're in a bubble where we haven't adopted it. Most people applying to work for us with the skills and experience we need use these tools as most companies are using it. The concerns we have about it aren't much different from supervising junior developers.

@GrapheneOS @dasmatus sure, I haven't personally had to deal w/ a whole new generation of devs being brought up w/ these tools or mid-to-senior devs forced to use LLMs, much less having to hire + train them.

I sympathize w/ your plight, and am trying to find ways to insulate folks from this bubble. including finding legit alternatives that can fulfill the need for LLMs (perceived or real), w/o contributing any support to big-tech LLM firms

@elle @GrapheneOS @dasmatus

What options are available?
What are the rough costs involved?
What kind of expertise is needed to help?
What is a realistic timeframe to shift from a big tech frontier AI tool to an alternative option?

I am genuinely asking, this is not to provoke.
I lack knowledge and I am trying to better understand the situation, I think you can help me (and others) to have a better picture.

In this moment, I really fail to see what's wrong in revising code with these tools.

@MagentaMold

re: alternatives, idk there are no perfect, 1-to-1 alternatives that are generally applicable. frontier open models may help, and they at least contribute (somewhat) back into the commons they used to train/derive the models. they also undercut the business model of the big-tech LLM firms, which is a good unto itself.

re: reasons not to use LLMs, you can start w/ https://txt.ltdk.xyz/sloppery-slope/

I've collected some other resources here: https://codeberg.org/tech-cooperatives/resources/src/branch/main/cail-resources.md

@GrapheneOS @dasmatus

@MagentaMold

the reason open models are imperfect is they still maintain the property of consuming FOSS + other data non-consensually, and w/o attribution. it still takes energy to run those models, so depending on the deployment, there could still be ecological harm.

though, there is more opportunity to host them in datacenters/hosting facilities that take true renewable energy + environmentalism seriously.

against the big-tech LLM firms spinning up gas generators

@GrapheneOS @dasmatus

@MagentaMold

there are also the cognitive decline effects, which still likely occur w/ open models (ur still using the same agents + tooling which lead to decline).

h/e, in a use-case like "scanning for bugs", this specific harm is reduced to be on the level of "ran coverity". as long as you're engaging in rigorous mental activity outside the scan, cognitive decline should be minimal.

@GrapheneOS @dasmatus

@elle @GrapheneOS @dasmatus

I couldn't read all the links, I will catch them up slowly.
I see most of them make the same point: code quality being lower.

I agree, with this reserve: what I get back from AI will be _very_ different from others would get, because I will pack my expertise in my prompts (which is not enough to ask proper code).

Great devs tho, will possibly have quality output.

On top of this, as long as strict review is in place, I would say that there's no real difference.

@elle @GrapheneOS @dasmatus

As per pollution and similar topics, people shouldn't blame a small company for using newer technologies.
They should blame technology
builders/owners for not building it or maintaining it in a sustainable way.

If I buy a car to go to work, I can't be blamed for the entire automobile pollution of the world, can I?

The "if nobody use it" argument is flawed, because you cannot educate/enforce this decision planet-wise.

@MagentaMold @dasmatus

> The "if nobody use it" argument is flawed, because you cannot educate/enforce this decision planet-wise.

mb, but we *can* call out & criticize projects we respect for using it.

> They should blame technology
builders/owners for not building it or maintaining it in a sustainable way.

I do, loudly & regularly. I can do both, imagine that

@GrapheneOS does not have the most terrible AI/LLM policy. given their goals, I would expect them to have an anti-AI stance

@MagentaMold @dasmatus

I understand @GrapheneOS take that using AI/LLMs for bug finding is useful as a defender, especially w/ cop/mil-tech using it to find exploits. I've already made my case for exploring alternatives to proprietary models from the most harmful companies, I won't repeat them again.

again, I'm not singling out GOS as some sole bad-actor, this plague has infected so much of the FOSS space. it's just sad to see another solid project take a middling stance on big-tech LLM use

@elle @MagentaMold @dasmatus The reaction towards us is highly disproportionate to what you describe as a middling stance on it.

We're very interested in running open weight models ourselves and eventually open source models. We've even been working on making our own open source text-to-speech models.

We decided the standard for what's shipped in the OS is that it needs to be open source and we can provide open weight models as packages in our App Store instead of bundling those in the OS.

@GrapheneOS I'd be curious to see an Article or doc that kinda shows how a dev's AI environment is setup and maybe an example that shows how it's being used.

@GrapheneOS I disagree with you on this, without attacking you:

> The concerns we have about it aren't much different from supervising junior developers.

If you tell a junior developer that/where he or she made a mistake, he/she will learn and not do it again.

If you tell a LLM that/where it made a mistake, it will output "You're right, I shouldn't have done this" … and do it again. There is no learning from feedback.

With a human junior developer, you know what mistakes junior developers commonly make … what *you* struggled with at the beginning of *your* career … you know the common pitfalls to look out for. The mistakes of a human junior developer are obvious to more experienced developers.

What a human junior developer wouldn't do is write code that looks flawless to a senior engineer *and* to all the configured linters/tests but contains an extremely subtle bug that goes unnoticed despite careful code review. "AI" coding assistants do that, however.

Similiar to how AlphaGo plays Go in a way no human has ever played Go before, "AI" coding assistants write code in a way no human writes code, and "AI" coding assistants are thus much more likely to introduce bugs than humans.

Your concerns should be very different from supervising junior developers.

@davidculley

> If you tell a junior developer that/where he or she made a mistake, he/she will learn and not do it again.

Definitely not our experience.

> There is no learning from feedback.

It can be kept as context.

> code that looks flawless to a senior engineer *and* to all the configured linters/tests but contains an extremely subtle bug that goes unnoticed despite careful code review

Projects with careful review, static analysis, dynamic analysis and tests still introduce many bugs.

@GrapheneOS right, and one of my points was just imagine if all this compute & resources had been committed to, for instance, funding the Rust-for-Linux devs, the in-kernel testing infra, and paid community reviewers/maintainers (not corpo-backed devs).

it doesn't address the immediate practical need to address the fallout of LLM-found bugs.

@dasmatus

@elle @GrapheneOS @dasmatus We don't live in a world of imagination.

@prism my apologies, swap out "just imagine" for "consider for a moment" or "just think"

@GrapheneOS @dasmatus

@elle @GrapheneOS

The hardware security requirements GrapheneOS has for device support has nothing to do with this. GrapheneOS is a privacy, security, and usability project. The requirements for device support and using AI code review to find exploits both serve to further these goals.

@HybridStaticAnimate

"The requirements for device support and using AI code review to find exploits both serve to further these goals."

how does using tools from outspoken fascists who have the explicit goal of eroding privacy, expanding the surveillance state, targeting dissidents + immigrants, etc. further the goals of a security + privacy project?

if you absolutely insist on using LLMs, mb choose ones not developed by fascists. mb some of the open models fit that description

@GrapheneOS

@elle @GrapheneOS

It finds exploits that were otherwise unknown, which can then be patched. If they can be found and exploited by attackers, they can be found and patched by defenders. Youre asking them why they would dare to use a hammer for a nail when hammers are used to hurt people all the time.

@HybridStaticAnimate

no, I'm asking why use the hammers w/ a swastika on the label, instead of alternative hammers.

again, this isn't an attack specifically on @GrapheneOS, Linux (specifically Linus) defense of "we're using the nazi tools, don't like it? fork off" is much worse, especially since the kernel is a more foundational technology.

but saying, "hey we're not the only ones doing bad shit" isn't exactly great, either.

collaborating on finding ways for us all to do better is good, tho

@elle @GrapheneOS

It... does not have those symbols. We have been using this technology for decades. It only took off recently because there is enough processing power to handle bigger tasks (even if those tasks are generating slop, people chose to make it do that, which is an abuse of the technology). The greater issues with AI were bound to happen with something else if AI did not take off. The issues are far deeper, socially, economically, politically, and AI is a symptom of it.

The tool is just a tool, not associated with any negative imagery. Its abuse is purely the product of humans, and GrapheneOS is using it to achieve some good.

Dont get me wrong, I am incredibly anti AI, and I would rather GOS locally host and run these models, but that will take time.

@HybridStaticAnimate

I was writing metaphorically... yes, none of the frontier AI labs are stupid enough to *literally* use a swastika as their logo. but one very prominent one has a leader *bold* (read incredibly stupidly asshole-ish) enough to do *two* nazi salutes at the U.S. presidential inauguration. so xAI's Elon Musk is not that far from stamping a swastika on his forehead, is he?

@GrapheneOS

@HybridStaticAnimate

both Anthropic and OpenAI are also gleefully aiding the genocide in Gaza + Lebanon, and automated bomb strikes in Iran. notably a girls school w/ IRGC family members in attendance (little girls). all w/ the "cover" of it being old info in the training data...

these are all outright fascists, who have also admitted they're fine with the non-negligible prospect of AI killing all of us. probably marketing, mb true believers. either way, fuck them into the sun

@GrapheneOS

@elle "LLM fascists" lmao

@GrapheneOS