GrapheneOS imposes a set of restrictions on how generative AI can be used in our project and community. It isn't forbidden but rather we imposed rules based on actual issues.
Posts made in our forum, chat rooms, issue tracker and the rest of our project must be written by humans. There's an exception from this rule for translation and accessibility features. That means commit messages, pull request messages, bug reports and feature requests are expected to be written by humans, not a machine.
Replying to @GrapheneOS@grapheneos.social
@GrapheneOS for an OS project that refused porting to non-PIxel h/w for so long b/c other platforms didn't meet security reqs, this pro-LLM stance is ludicrous.
~all bugs found by LLMs can likely be found by other methods using the same-or-less compute resources, especially w/ the human effort "babysitting" the LLMs to ensure they don't go off the rails.
there just aren't billionaire fascists subsidizing that compute, be honest about it.
stop covering + doing PR-by-action for LLM fascists
Replying to @elle@weathered-steel.social
@elle @GrapheneOS what's the relation of Google being Google and a language model?
Replying to @dasmatus@mastodon.social
are you serious? what is the relation b/w GrapheneOS taking security so seriously that they wouldn't port their OS to h/w that didn't have Pixel's special sauce, and now being fine using LLM code + bug reports + dedicate human time to "babysit" the LLMs?
my intention is not to "bully" @GrapheneOS. we need folks like you in FOSS building rock-solid projects. I'm merely voicing criticism, and hoping the team explores other alternatives.
leave big (proprietary) LLMs in the bin
Replying to @elle@weathered-steel.social
@elle @dasmatus GrapheneOS is based on Linux, AOSP, Chromium and hundreds of other upstream projects heavily using AI generated code.
Our OS development team hasn't been using any AI code generation but yet the OS is heavily using it because the vast majority of the code comes from upstream projects. Linux and Chromium are both massive projects.
Firefox is heavily using it too, so that isn't relevant to this. It's nowhere close to what we would need to use it as a replacement regardless.
Replying to @GrapheneOS@grapheneos.social
@elle @dasmatus Our policy is only accepting high quality code. We've always had to do careful code review and avoid trusting submissions.
We aren't happy with the flood of slop which gets submitted to us via our issue tracker, security@ email and pull requests. However, a lot is reporting or fixing valid issues.
For pull requests fixing valid issues, we can either get them to improve it to meet our standards or do it ourselves. We often start with rounds of review but finish it up ourselves.
Replying to @GrapheneOS@grapheneos.social
@elle @dasmatus In the past couple months, there has been a massive flood of valid vulnerabilities being discovered by frontier AI models. Nearly all of those are from human written code, often from many years ago.
Linux has heavily shifted away from adding new features and improving performance to fixing the bugs found by these models. They're being forced away from sticking with C and a monolithic kernel design. It's on ongoing crisis and people can regularly grab working exploits off GitHub.
Replying to @GrapheneOS@grapheneos.social
@elle @dasmatus Cellebrite and other exploit development companies are having growing success exploiting all kinds of targets. They're openly talking about how they're using AI tools for reverse engineering, vulnerability discovery and exploit development.
It's a very large and rapidly growing short term advantage for attackers. The consensus among security researchers is that it's short term because vulnerabilities are being rapidly found by defenders too including before merging or release.
Replying to @GrapheneOS@grapheneos.social
@elle @dasmatus We have a responsibility to protect our users. We're aware frontier AI models can be used to find many vulnerabilities. We made several accounts on cheap plans with a fixed amount of usage, applied for and received cybersecurity access and began testing it. We've found what others have which is that these tools find many actual vulnerabilities including subtle domain specific issues. It's largely beyond the scope of what traditional static analysis tools even attempt to do.
Replying to @GrapheneOS@grapheneos.social
@GrapheneOS @dasmatus I understand, what I'm suggesting is that we as a community explore other solutions, possibly involving open models (hopefully not), that can also provide serviceable results for vuln disco.
for example, taking a symbolic + static analysis, and fuzzing approach that shells out to an open model LLM (or dev a fine-tuned small model) for path exploration when the other tools get stuck.
in other words, use/develop purpose built tools instead of rigging a GPT
Replying to @GrapheneOS@grapheneos.social
@elle @dasmatus As with anyone else, we make mistakes and there are many things we don't know about. There are undocumented behaviors of library and Linux kernel APIs. We have the wrong mental model about how some things work. We can review a pull request a dozen times with great care and still miss things quickly discovered with one these tools. We confirm it's a real issue and figure out how to fix it ourselves. Our fix can also be checked. We write small volumes of code so it's very cheap.
Replying to @GrapheneOS@grapheneos.social
@elle @dasmatus Our OS development team and most of our other developers aren't using AI coding assistants in the form of an IDE. They'd need to be run in a VM as a secondary workspace with a separate GitHub account or a similar approach.
We're in a bubble where we haven't adopted it. Most people applying to work for us with the skills and experience we need use these tools as most companies are using it. The concerns we have about it aren't much different from supervising junior developers.
Replying to @GrapheneOS@grapheneos.social
@GrapheneOS @dasmatus sure, I haven't personally had to deal w/ a whole new generation of devs being brought up w/ these tools or mid-to-senior devs forced to use LLMs, much less having to hire + train them.
I sympathize w/ your plight, and am trying to find ways to insulate folks from this bubble. including finding legit alternatives that can fulfill the need for LLMs (perceived or real), w/o contributing any support to big-tech LLM firms
Replying to @elle@weathered-steel.social
What options are available?
What are the rough costs involved?
What kind of expertise is needed to help?
What is a realistic timeframe to shift from a big tech frontier AI tool to an alternative option?
I am genuinely asking, this is not to provoke.
I lack knowledge and I am trying to better understand the situation, I think you can help me (and others) to have a better picture.
In this moment, I really fail to see what's wrong in revising code with these tools.
Replying to @MagentaMold@mastodon.social
re: alternatives, idk there are no perfect, 1-to-1 alternatives that are generally applicable. frontier open models may help, and they at least contribute (somewhat) back into the commons they used to train/derive the models. they also undercut the business model of the big-tech LLM firms, which is a good unto itself.
re: reasons not to use LLMs, you can start w/ https://txt.ltdk.xyz/sloppery-slope/
I've collected some other resources here: https://codeberg.org/tech-cooperatives/resources/src/branch/main/cail-resources.md
Replying to @elle@weathered-steel.social
the reason open models are imperfect is they still maintain the property of consuming FOSS + other data non-consensually, and w/o attribution. it still takes energy to run those models, so depending on the deployment, there could still be ecological harm.
though, there is more opportunity to host them in datacenters/hosting facilities that take true renewable energy + environmentalism seriously.
against the big-tech LLM firms spinning up gas generators
Replying to @elle@weathered-steel.social
there are also the cognitive decline effects, which still likely occur w/ open models (ur still using the same agents + tooling which lead to decline).
h/e, in a use-case like "scanning for bugs", this specific harm is reduced to be on the level of "ran coverity". as long as you're engaging in rigorous mental activity outside the scan, cognitive decline should be minimal.
Replying to @elle@weathered-steel.social
I couldn't read all the links, I will catch them up slowly.
I see most of them make the same point: code quality being lower.
I agree, with this reserve: what I get back from AI will be _very_ different from others would get, because I will pack my expertise in my prompts (which is not enough to ask proper code).
Great devs tho, will possibly have quality output.
On top of this, as long as strict review is in place, I would say that there's no real difference.
Replying to @MagentaMold@mastodon.social
As per pollution and similar topics, people shouldn't blame a small company for using newer technologies.
They should blame technology
builders/owners for not building it or maintaining it in a sustainable way.
If I buy a car to go to work, I can't be blamed for the entire automobile pollution of the world, can I?
The "if nobody use it" argument is flawed, because you cannot educate/enforce this decision planet-wise.
Replying to @MagentaMold@mastodon.social
> The "if nobody use it" argument is flawed, because you cannot educate/enforce this decision planet-wise.
mb, but we *can* call out & criticize projects we respect for using it.
> They should blame technology
builders/owners for not building it or maintaining it in a sustainable way.
I do, loudly & regularly. I can do both, imagine that
@GrapheneOS does not have the most terrible AI/LLM policy. given their goals, I would expect them to have an anti-AI stance
Replying to @elle@weathered-steel.social
I understand @GrapheneOS take that using AI/LLMs for bug finding is useful as a defender, especially w/ cop/mil-tech using it to find exploits. I've already made my case for exploring alternatives to proprietary models from the most harmful companies, I won't repeat them again.
again, I'm not singling out GOS as some sole bad-actor, this plague has infected so much of the FOSS space. it's just sad to see another solid project take a middling stance on big-tech LLM use
Replying to @elle@weathered-steel.social
@elle @MagentaMold @dasmatus The reaction towards us is highly disproportionate to what you describe as a middling stance on it.
We're very interested in running open weight models ourselves and eventually open source models. We've even been working on making our own open source text-to-speech models.
We decided the standard for what's shipped in the OS is that it needs to be open source and we can provide open weight models as packages in our App Store instead of bundling those in the OS.
Replying to @GrapheneOS@grapheneos.social
@GrapheneOS I'd be curious to see an Article or doc that kinda shows how a dev's AI environment is setup and maybe an example that shows how it's being used.
Replying to @GrapheneOS@grapheneos.social
@GrapheneOS I disagree with you on this, without attacking you:
> The concerns we have about it aren't much different from supervising junior developers.
If you tell a junior developer that/where he or she made a mistake, he/she will learn and not do it again.
If you tell a LLM that/where it made a mistake, it will output "You're right, I shouldn't have done this" … and do it again. There is no learning from feedback.
With a human junior developer, you know what mistakes junior developers commonly make … what *you* struggled with at the beginning of *your* career … you know the common pitfalls to look out for. The mistakes of a human junior developer are obvious to more experienced developers.
What a human junior developer wouldn't do is write code that looks flawless to a senior engineer *and* to all the configured linters/tests but contains an extremely subtle bug that goes unnoticed despite careful code review. "AI" coding assistants do that, however.
Similiar to how AlphaGo plays Go in a way no human has ever played Go before, "AI" coding assistants write code in a way no human writes code, and "AI" coding assistants are thus much more likely to introduce bugs than humans.
Your concerns should be very different from supervising junior developers.
Replying to @davidculley@hachyderm.io
> If you tell a junior developer that/where he or she made a mistake, he/she will learn and not do it again.
Definitely not our experience.
> There is no learning from feedback.
It can be kept as context.
> code that looks flawless to a senior engineer *and* to all the configured linters/tests but contains an extremely subtle bug that goes unnoticed despite careful code review
Projects with careful review, static analysis, dynamic analysis and tests still introduce many bugs.