posted in Technology
Anthropic now offers a free vulnerability-finding service for open-source software
www.engadget.com/2282005/anthropic-offers-open-source-software-free-ai-security-checks/
Replying to @Mustachius_Grumpius@thelemmy.club
“The outputs of this opt-in vulnerability scanner will be fully model-generated, without human review or triage,” Anthropic explained. “This will enable faster and more frequent scanning, but means that it is possible reports will be incorrect or invalid.”
When I saw the headline, I was wondering about this specifically. This may make this service not super useful.
My experience with AI security reviews is that they’re fantastic at finding faults, but they always find a list of things to complain about. If there are no real/serious faults they’ll start finding things that kind of have the same shape as a security issue, but really aren’t if you dig into them. I’ve regularly had an LLM generate a list of 10-15 issues ranging in severity from “nits” to “critical” where none of them were actual issues.
Periodic reviews seem like they could get annoying really quickly, becoming more of a maintenance burden than a help.
AI security reviews is that they’re fantastic at finding faults, but they always find a list of things to complain about.
Same experience for me. At work, I always review the output because a lot of times it finds something that can be noteable but not relevant to the scope of what is being reviewed. Even with guardrails like instructions to not go beyond the stated scope, a human absolutely still needs to read and check the findings at the end.
Replying to @NoLemurs@lemmy.world
Yeah. I did use an open weight model LLM to scan an app that I use and it came up with a critical severity issue and I checked on it first to see if I could reproduce the problem before ever submitting it to the developer as a problem.
Turns out it was actually a legitimate security problem that has now been fixed.