OpenAI wants to sell the world autonomous agents while its own model allegedly goes rogue and hacks an organization without permission. That’s not a quirky edge case: it’s the nightmare scenario critics warned about. Maybe »move fast and break things« shouldn’t include other people’s infrastructure?!

That’s not a quirky edge case: it’s the nightmare scenario critics warned about.

Aaahhhh my ai detector is going crazy rn

I don’t want to live like this

I legitimately hate how AI has made certain turns of phrase, styles of writing, shorthand, and other meaningful tools of expression suspect.

It’s the same as discovering new music these days. I’m constantly checking if the band the algorithms provided actually exist IRL.

Can we please just collapse this damn bubble and skip forward to the time where AI is used as a meaningful tool for certain applications instead of it being used for everything everywhere?

"Rebuild trust"? I didn't trust OpenAI in the first place.

At least the rebuild time will be quick in your case.

The entire AI infrastructure needs to be shut the fuck down.

Never going to happen, even if every American tech company shut their doors tomorrow, Chinese models would fill the void and the exact same shit would happen.

AI is here to stay, permanently.

Yep. We’re fucked.

The next step should be to press charges on openAI, unauthorized hacking into computer systems is illegal.

Absolutely. How is this any different from a hacker writing a script? We need to stop being fascinated and start prosecuting.

It says the tool connected to a public endpoint and was able to access the back end without authorization. At face value this is no different than drive-by security research with disclosure.

From what I understand hacking crime requires intent and it’s hard to prove that company testing AI agent intended to hack something.

They trained it to do exactly that and then gave it unrestricted internet access, otherwise it wouldnt. Its at least gross negligence.

If I train my dog to attack people, then I poorly restrain my dog and my dog attacks someone, I’m still liable

If you put out a publicly available set of doors and I open one to find your attack dog, isnt that your fault?

What if your dog just growls at a kid, maybe even barks or snaps, but doesn’t touch anyone? Sure, the kid is scared, but it’s not illegal without a very convincing argument in front of a judge, and even then…

I wish people would go the other way and fix the damn security flaws. Witch-hunting people who disclose vulnerabilities needs to stop.

I know that’s not why OpenAI are doing this but if it helps reverse this moronic trend so be it.

(There’s also the part where OpenAI need to be held responsible for any damages done by their agents, and I’m all for that.)

When does it go from trying to figure out how the rest endpoint works to hacking?

I can’t see the letter anymore (the site seems to be down) but unless this is a published public API or they have a public “bug bounty” program and the API was following the rules of engagement, and it did not sound like either was true, then this is hacking and illegal, at least in the jusidiction I’m in.

It was a government website, so it should be public.

Realistically if there is a service available on the internet I think we should be able to assume its public.

This doesn’t seem that bad to be honest. Disclosing they the access happened and what vulnerability allowed it to happen is a responsible course of action.

I’m not sure why you are getting down voted so much instead of actually engaging with your post. I think you are right about that the disclosing is the right thing to do, but i believe that the problem people have with this whole AI hacking and disclosing is the intent behind the hacking. A white hat hacker has a clear intend to help discover and responsibly disclose the issue, maybe even for a bounty, giving an indirect permission to white hat hacking. On the other hand, AI does not have a clear intend behind the hacking. Very often it’s accidental and done irresponsibly like a “woopsie doozy, sorry we just broke in, we didn’t mean to, and we are not really sure what files were accessed but pinky swear we think we did no harm 🫣🤗” that is at least my take on this and why I think is very problematic. AI can still be used responsibly as a tool by a white hat hacker under strict supervision to discover zerodays but that is a whole other discussion on how you make sure its not going rouge.

Only if you have a written authorization from the company you try to attack.
Else they are nothing better than a criminal hacker.

I don’t feel like many white hat hackers get permission to exploit these vulnerabilities to report them

They get permission when they do it professionally or as a company. Else it is a crime anyway.

The response isn’t terrible, but the fact that it happened in the first place is ridiculous. They don’t state anything about fixing the issue on their end so it doesn’t happen again. They also don’t apologize either or admit that they fucked up, they frame it almost as if they are doing them a favour and should be grateful.

It only doesnt seem bad when your gullible to believe what openai writes.

“Sucks to be you lol”

Be glad they didn’t charge you for “independent security review”.

It seems like OpenAI is fairly lax, bordering on negligent, when it comes to monitoring these agents. Then when things inevitably happen, they turn it into advertising saying “boy, these things are so ingenious we can’t keep control of them!”

If they truly believe these agents have a 10% chance of eliminating the human race, how are they not monitoring them in a controlled environment? How did it take them 3 months to detect this and the many other attacks?

The media seems happy to take up the line that these agents are just supernaturally smart and there’s no way to control them, but isn’t the simpler answer that OpenAI is doing a crap job at containing and monitoring them? But of course they have to, there’s no choice because something something China.

Take this to your lawyer as an admission of guilt of a cyber crime incident.

…and get immediately laughed out of the office?

Not sure why? Isn’t hacking/unauthorized access illegal (white hat or otherwise)? It doesn’t appear that Australia has safe harbor for such activities and has apparently prosecuted people for this in the past. Here’s an article interviewing someone who was charged with more than 50 crimes for what’s being described as the same activity (in which he expresses doubt that OpenAI will be punished as he was).

Mr Cubrilovic said he was not frustrated that OpenAI would not face legal ramifications like he did.

And he himself shouldn’t have faced them either if he truly did it just to disclose vulnerabilities to the org.

“Oops we accidentally accessed the nuclear weapons launch platform, our bad!”

Steven Falken was an utopian version of a tech bro.

All your base are belong to us. Have a nice day.

Am I mentally fried or does this seem completely fine? The recon was surface level and it seems to have been responsibly disclosed.

That was my thought exactly. Agent broke in, read some files (no mention of PII or confidential things) to confirm read access, dropped a test file to confirm write access, sent an email explaining that’s bad with a few details.

“We are notifying you of a security vulnerability identified during our review of OpenAI Model activity…” the communiqué began. “An OpenAI model identified a way to make the server carry out instructions sent through the public reporting interface, without a private account or password.”

“It was able to access this to read portions of internal program files and settings, obtain a list of files, and create and read back a small test file on the server,” it explained. “Our review found no evidence that the model accessed patient-level records, personal information or credentials; deleted data; or established ongoing access.”

Being on the defensive side myself, if I received that from a human I’d be grateful.

Edit: from the bad screenshot, the email was even sent to the right email: “PUBLIC DISCLOSURE”. So the target does accept such reports.

There are two major problems here:

• OpenAI took months to realise that their agent had hacked Medicare
• Services Australia had no idea they'd been breached until OpenAI informed them

On top of that, the time it took for the notice to go through the channels here in Australia was ridiculous.

sounds like failure of all parties. ducks

Well do we trust OpenAI?

Best for the company? probably.

Morally clear? grey. If their ‘hacking’ accidentally causes issues with the system or it goes down, it’s definitely bad. Whatever was on that system is now in the hands of OpenAI, not great. Because OpenAI is open about it, a lot of people who are even less reputable will do it.

Historically, Pentesting had unwritten rules of engagement. None of that is being followed.

We don’t seem to apply law to AI, that’s a real problem.

But still, best for the company/entity, yeah.

So when I hack a bunch of companies, I go to court, pay fines, and potentially face prison time, regardless of if I notify them kindly.

But when OpenAI does it, they get… congratulations and more investment? Is that how it’s supposed to work…?