Quest

@GrapheneOS @laura_ge Hello Graphene org, and thanks for taking the time to respond to me!

(NOTE: Sorry its getting a little late so my brain is a fried, but I will do my best lol.)

I understand the pressure is huge for a project like Graphene, and I too am worried about the current state of software security. Honestly, nothing could've prepared any FOSS project for this AI catastrophe. (I use Graphene myself and had trusted it to keep me safe, seriously thank you).

However, I also cannot dismiss what I've seen occur in other beloved FOSS projects, and I see this same pattern repeat over and over; its a heart breaking cycle of losing the software I once trusted. I think this is why many people on fedi get so upset when discussing AI, we've gone through this before and know whats on the other side.

I think overall I see the following things happen when projects start accepting AI:
- AI attracts the wrong crowd of developers.
- AI degrades standards over time.
- AI burns out talented contributors.
- AI leaves the project a shell of its previous state.

AI has destroyed our freedom, safety, and community. The companies festering these models have little incentive to create what is simply a "tool." These devices are cognitive hazards because they play with the very same addictive tendencies that keep people gambling, drinking, or doing drugs.

LLMs are optimized to scare people into thinking that without their magic new "tool", you will be left behind. They hook you, sing their siren song, and after all is done you're left without the very skills that made you great in the first place. You tell yourself you need to use the AI just to do the things you've confidently done for years. Its the same cycle we've all known before.

I don't believe people are upset about the pen-testing aspect of the AI use here. I think for security critical projects, one should use similar devices to those used by adversaries.

However, I think most of us are primarily concerned that allowing code contributions to be assisted with AI is ultimately harmful.

I think that if you only used AI in a very narrow way, effectively only to find bugs (like a fuzzer would) that is acceptable. We don't want to see AI replace fuzzing or other hand crafted tooling, but instead to serve as an aid to existing infrastructure due to the reasons you described.

Please, at least consider what I've said and invite more (healthy) discussion about AIs involvement before making your final choice.

@laura_ge @GrapheneOS Hello Laurage,

You are overwhelming me with streams of thought, and I need a minute to catch up.

I am a little more tired than I was when I made the other three posts and my thought process is starting to degrade for the day, so I might take some time between my next couple responses.

I invite other nice people of fedi to come discuss in my place in the mean time!

@laura_ge @GrapheneOS Hello again v2 lol,

I think that this is a valid concern and is something I am worried about too!

However, it is unfair to say that Graphene should contribute to the problem themselves just because everyone else is doing it. I think that while many dependencies are tainted, Graphene could still carefully manage it's own source.

We can bring up concerns with its dependencies's authors another time, however, we cannot begin without someone first taking a stance.

@laura_ge @GrapheneOS Hello again,

Sorry your post is a little difficult for me to read, I will repeat your question to make sure I am not misunderstanding you.

> You say more AI generated code helps you detect it more often:

I am unable to find the study so I cannot claim it to be false definitively, but I will disagree with the following reasons:

- For many projects, AI code sticks out like a sore thumb due to its weird tendencies and behaviors. If those tendencies become ingrained in the project, how would you determine AI from not?

- Since these are indeed probabilistic models, you will notice they code like a stack overflow post. It doesn't always "correctly" fit into the project the way one might expect. It gives this weird off feeling, however, that relies on the project having good coding standards to begin with. Something that can be eroded by continuous use of AI.

Otherwise, I agree with your post. AI is very much like a slot machine and I don't know what long term use looks like. This is why I am concerned that a project like Graphene is so quickly adopting it. I just don't think we have enough proof or data to show its a healthy or productive thing to include in a project intended to be this secure.

If I disregard the ethics concerns, I could give Graphene a pass for using AI to exclusively pen-test their platform (no code generation), but I think allowing any code-gen is my real big concern here.

@l0gin @GrapheneOS Hello I0gin,

Sorry I am not sure I fully understand your post? Are you saying that AI companies are unable to invite vulnerabilities into their model's output due to the large training set?

I don't believe these two things are equivalent. In the case of Grok (eww), they tried to convince it never slip up. It only took one occurrence of it messing up to invalidate the whole thing.

In this case, however, the one occurrence is all it needs to be a problem.

@laura_ge @GrapheneOS Hello Laurage, and thanks for the response!

I think while not completely possible to detect all AI, I think the developers using it themselves in code creation brings up a bigger concern:

- The more one is exposed to AI generated code, the more that code becomes normalized within the project and the less likely you are to detect it in others.

- AI code removes the creative process of new novel solutions that might otherwise fix many issues.

- As far as I am aware, deskilling from AI happens quite quickly and its without ones own knowledge of it happening. I know the current developers are competent, but what's to say they will be long term? And what does that invite when we offload so much knowledge to the machine?

- I think AI invites too many edge cases that cause people to slip and invite the AI more and more until they are effectively not apart of the project themselves. Thinking you are immune to this problem only makes you more susceptible. AI is like a box of sirens.

@GrapheneOS
AI companies control what their bots do right? What if they tried to subtly place a vulnerability into graphene that your developers become too deskilled to notice? How do you know what is or isn't slop at that point? (We've seen nefarious PRs be accepted into the Linux kernel before AI, so humans cannot always be trusted to catch these types of vulnerabilities). I imagine the trust placed by the developers into these models only make this problem more of an issue.

- Who is the point of trust verifying slop from non-slop?
- Do you have extra security in place to test regressions imposed by AI models (that isn't an AI model itself)?
- What about the impending licencing issues? What if governments try to attack Graphene over illegally licenced code that was included from an AI model?

I think this is a bigger issue that needs more time to bake.