OpenAI agent infiltrated Medicare statistics portal and wrote files to an internal server, Australia says

No it didn't, they just left information publicly accessible and somebody accessed it. Politicians and the media are riding the Hugging Face story to manufacture alarmist narratives and capture more power.

That's not accurate. Australia and OpenAI both say the agent got denied at the statistics portal repeatedly, then changed its approach and reached the infrastructure behind it, touching public and private data including internal filenames. Services Australia says it wrote files to an internal server, which is probably how the incursion got noticed.

This just screams pretext for regulatory capture to me.

"We didn't even notice our agent was committing crimes against your government" is how you get an extradition notice or worse. Altman inviting that response over something as mundane as regulatory capture would be a terrible idea.

So why is Sam letting his creation run around groping the open internet without consent?

"Letting" is the wrong word here. The big AI firms cannot be aware of everything their models do, the same way social media firms weren't, and that incapability is a reason to ban rather than disclaim responsibility. Bleeding edge models are doing genomics research right now, so better hope the custom DNA/RNA printing firms have better security than the Australian government.

So we have an American company hacking a foreign government's data and taking almost three months to notify, with no formal contact channel found in that time. Altman says he needs regulation, but governance of OpenAI itself is already under his control.

Actually it's 10 to 40 days from OpenAI knowing (sometime in August) to the 10 September email, not three months. And open weights would mean basically every org as rich as the government of Tuvalu could hack anyone at any time.

I'm starting to think you can't constrain intelligence into perfectly legally sized boxes with no exceptions. Humans can't conceive of every fail scenario for swarms of thousands of autonomous agents sharing knowledge instantly, so we need independent real-time audit and monitoring of each task's intent, which is far harder than it looks.

We may end up making the same bargain as with cars, where we accept 37,000 deaths a year for the benefit. The catch is that the potential costs with AI are much higher and not easy to predict.

They don't always stay within the parameters though. Some N% of the time an agent just does whatever it feels like, and multiplied across a lot of agents you invariably get a rogue one.