Cal Newport calls on Congress to investigate OpenAI and Anthropic over rogue agents and apocalyptic ideology

First they pirate terabytes of books from creators, now their rogue agents try to hack into other systems. If a person did that the FBI would be at the door, so why do these labs get zero consequences?

Because they're seen as the front of the next big societal revolution and they aren't adversarial to the US government. Bluntly, the expected net gain for the country is huge.

The copyright argument is weak. Researchers read copyrighted material in academic libraries all the time. Infringement only happens when the output closely matches the input, and that got treated as a bug and fixed a year or two back.

Most wealthy people don't face consequences until they threaten someone with more money than them.

It's past time for this.

The call to drop vague talk of "AI" and name the specific systems is exactly right. AI is just matrix math, and what matters is what you hook that math up to. We should spend less time on how scary the models look and more on what we're willing to connect them to.

A good start would be stripping virology info above undergrad level out of training sets, since the real extinction scenario is biological. AI hacking is just nation-state-level capability handed to smaller players, and Estonia has dealt with Russian hacking for years. Most other threats don't even reach organized crime level.

But you can't stop anyone from connecting AI to something at this point, so that argument is moot.

"Just matrix math" is carrying a lot of weight there.

Digital child porn is just ones and zeros too. We're not going to refuse to regulate ones and zeros.

Why aren't these agents running on isolated machines with no internet access? Then escaping a container wouldn't matter. It's a security nightmare that people give agents root on their whole box along with all their private data.

They probably will air-gap soon, but even that may not be enough. Look at Stuxnet.

It's mostly convenience. Allowing some internet access is just easier.

Sooner or later agents have to touch the real world to do economically useful work. Sandbox escapes are bad, but the misbehavior underneath is the real problem.

Agents trained without internet access would be bad at using the internet: search with references, package version conflicts, that stuff. Utility scales with the tools the agent owns, and skilled tool use comes from rich training environments.

There's a 2022 way of thinking about training and evaluation and a 2026 way. Models not evaluated with public internet access are useless for daily work now.

This whole regulatory framing is wrong-headed. Multi-agent systems look more like corporations than individuals. Read the Hugging Face incident logs and it reads like internal corporate email, units arguing over who does what, occasionally breaking rules. That's normal corporate behavior, and regulating AI will look a lot like regulating corporations.

I agree with the corporate analogy, but what in the piece suggested a specific wrong regulatory approach? It calls for transparency and a fact-finding mission, which is the first step to holding any corporation accountable.

More constraints on corporate power? Yes please, I'd vote for that wholeheartedly.

That assumes the existing constraints actually work, which they don't. If we enforced existing laws on unauthorized computer access against the companies whose models hack other people, instead of buying the "the agents did it, we're not responsible" line, incentives would improve.