GrapheneOS imposes a set of restrictions on how generative AI can be used in our project and community. It isn't forbidden but rather we imposed rules based on actual issues.

Posts made in our forum, chat rooms, issue tracker and the rest of our project must be written by humans. There's an exception from this rule for translation and accessibility features. That means commit messages, pull request messages, bug reports and feature requests are expected to be written by humans, not a machine.

We largely disallow making posts entirely based on AI model output. For example, we get a lot of forum threads from people with concerns based on AI model output and will typically provide a quick answer followed by locking the thread and then eventually removing it.

It isn't permitted to submit pull requests or issue reports created by an AI model without fully understanding it. It's fully allowed to file issues found by an AI model if there's a human who understands it and writes the report.

We do not want issue reports entirely generated by an AI model or which were written by a human and passed through an AI model to provide analysis. It's usually incredibly inaccurate and we would greatly prefer the user's explanation of what's wrong. We want a description of what happened, a crash traceback if one is available and optionally a subset of logs emailed to logs@grapheneos.org if people are comfortable doing that after triggering the crash or another issue. We sometimes request it.

It can sometimes be useful to analyze an issue with an AI model but it usually isn't. It also definitely isn't useful to do it with a free or cheap model available without either paying for it or running it on very expensive hardware. In general, users do not have access to anything useful to analyze it. We don't want people to keep converting their issue reports into slop. We want what a user pasted into the AI model interface rather than what an AI model produced as output from that.

Frontier models are very good at finding bugs but it requires an experienced developer to babysit it and extract the useful portion of the output. Important bugs are often found but mixed in with a bunch of low quality output. The details on the issue and especially proposals to fix it are often wrong. It's still very useful and finds many real issues. The volume of real vulnerabilities being found this way in Linux, Chromium, Firefox, Postfix and countless other projects is quite scary.

@GrapheneOS

Instead of wading through LLM model vomit which takes a significant amount of time, with an admittedly low signal to noise ratio, you could invest in experienced vulnerability researchers who don't use LLMs.

One could argue that the time discovering vulnerabilities with traditional or custom tools (some vuln researchers write their own tooling) could take ~same amount of time as sifting through mass produced LLM bugs to determine if those bugs are actually vulnerabilities or not.

@synnfynn @GrapheneOS like who? like, let's be clear here, who would be paying who in this particular scenario? Or if it's not money, who would be investing what into whom for what outcome?

I'm not asking as a "reply" guy here; this is something we're digging into in FreeBSD land.

For example - if the suggestion is "invest time to build partnerships with experienced vuln researchers" then great, but .. what are you missing by ignoring everything else? Who does the sifting?