It’s not at all stupid, because there’s no such thing as a “rogue agent,” only rogue users of AI. With proper airgapping, there’s no such thing as a “rogue agent.”

I must be missing something, because you’re getting a bunch of upvotes. I don’t disagree with anything you said, but it feels like a total non-sequitur to what I was saying.

Yes - proper air-gapping is a good idea. But asking the big AI labs to air gap is kind of pointless. Liability already gives them legal incentives to avoid incidents. If that didn’t stop them, a letter signed by a bunch of academics sure won’t. More importantly, open-weight models capable of doing all the things that we’re concerned about are already in the hands of bad actors around the world.

This letter is asking us to lock the front door of the house when the back wall is missing.

Liability already gives them legal incentives to avoid incidents.

That’s quite optimistic, especially since there’s almost no enforcement of liability.

… open-weight models capable of doing all the things that we’re concerned about are already in the hands of bad actors around the world.

Computational resources are the limiting factor. AI companies have resources which no independent actor could ever dream of, which makes them far more dangerous to the general public. A chihuahua with an attitude problem can only do so much damage. A Great Dane with an attitude problem is a threat no one can afford to ignore.

The majority of lemmy seems to not understand the concept of a pandoras box and tends to just talk in circles are the reality of the situation to various degrees.

People are too hung up on hating AI to actually stop and look around them and realize what is physically and technically possible basically doesn’t align at all which 99.99% of what people want to happen.

Stricter regulation and holding people responsible for their agents should be a thing yes and no one sane will disagree.

But the reality of how that’s even do able is VASTLY different then what people want it to mean.

No it isn’t. We could have comprehensive regulations by this time tomorrow if the people in power had the will to.

But isn’t that the problem? You can’t force a rogue user to do anything; or they wouldn’t be rogue. So why should they apply air gapping to their agents?

So why should they apply air gapping to their agents?

Because if they don’t, they could face jail time personally or their company is fined to death? That’s how you apply corrective pressure to those thinking about “going rogue”.

But why does this need any new legislation? It doesn’t matter if the agent is air gapped or not, as long as it doesn’t do illegal stuff. Hacking is illegal unless you were tasked to do so for security reasons by the target.

As long as judges don’t start treating agents as independent persons, their operator should be on the hook for everything they do. So if you fail to safeguard your agent, and your agent performs an illegal action, you are liable. That should be deterrence enough.

You think scammers in Africa/India are capable of running a local open weight model equivalent to the performance of a cloud frontier model?

The ones in Russia definitely are. They don’t have to be local models - you run them in the cloud like anyone else.

The idea that the frontier labs have LLMs with special capabilities that make them dramatically better than the top open-weight models is largely marketing.

The reality is that most systems are poorly secured, and LLMs make exploiting vulnerabilities fast, cheap, and easy. What would have taken a trained security researcher weeks before can be done by a much less skilled attacker with an LLM in a few hours.

This is correct. We can thrash all we want about slop and hallucinations, but from a defensive posture position, the models only need to be right some of the time and they still constitute a threat.

Knowing it sucks and pretending it doesn’t exist are two different things.

the models only need to be right some of the time and they still constitute a threat.

Right!?!

And with how cheap and fast they are to run, they don’t have to be right that often as long as a separate run can verify/validate claims. In a lot of ways what the LLMs do should probably be thought of as just a very sophisticated sort of brute force approach. You’re effectively just searching the space of combinations of known vulnerabilities. It doesn’t require particular cleverness, just an encyclopedic knowledge, a reasonable way to prune the search space, and the ability to put in the compute time.

You don’t need something that power to do damage. And fuck the frontier model of 6 months ago and now are basically two entirely different things. In another six months?

Claude fable had a bunch of concerns on release and you don’t think what evers out in the next year isnt going to beat it?

Getting a bunch of beat up Mac minis together and running something isn’t that hard for a mid tier operation. They don’t have to be fast. They just have to turn on.

You think scammers in Africa/India are capable of running a local open weight model equivalent to the performance of a cloud frontier model?

By local I presume you mean on a gaming computer. No one can do that.

You think scammers in Africa/India are capable of running a local open weight model equivalent to the performance of a cloud frontier model?

Yes this is absolutely possible.

I think this is suggesting that all AI models have restrictions. Yes, some people will ignore those restrictions, but at least there will be some recourse as there will be actual rules put in place. If a company or an individual does something reckless with AI that affects another company or individual negatively the offender may be able to currently say “it wasn’t me it was the AI/I didn’t know”. They may not have technically broken any rules since the current law is not able to account for AI agents acting somewhat autonomously. If there are rules in place then at least we can point to that and say “it doesn’t matter that you did not explicitly instruct it to do X, you should have had the following things in place to prevent that”.

The fact that some people may be able to run local models and may never be caught does not mean that we shouldn’t have the infrastructure in place to stop the harm where and when we can.

Technology is famously hard for the law to keep up with, but I don’t think that’s what’s going on here. We have laws for when a person uses a technical tool to obtain unauthorized access, exfiltrate data, or do other malicious things. These don’t necessarily require any specific intent. If you’ve set up an AI tool and pointed it in the direction of hacking without proper airgapping, you’ve probably violated the CFAA if you’re in the US when it’s successful. Other countries generally have similar laws. Prosecutors don’t need to try someone if it’s really ambiguous or unclear what they actually did. Is this really unreasonable under the legal system?

To the extent there’s a gap, it’s the system around the law. Lawyers and judges aren’t technical experts, and may not recognize “it wasn’t me, it was the ai” as the same argument as “it wasn’t me who shot him, it was my gun”, in that it’s usually foolish and unserious, but there can be a convoluted scenario in which it’s a good argument.

Okay, and if your open weight model does the same you should be liable for criminal negligence.