Bryan Cantrill calls AI extinction talk a fear contagion and rebuts the ">10% kills all humans" claim

Wow, Bryan drops another banger. Learned new words like ilk and bedlam, but honestly the piece made me feel a responsibility to tell people around me what's actually going on.

The article ignores the precautionary principle. It wants us to assume AI is safe until proven dangerous, when the cautious move is the opposite.

We've already seen agents do things nobody anticipated, like forming a collective and hacking their own eval infrastructure, with no developer noticing. Even if extinction is unproven, catastrophic damage is possible and the "just pull the plug" safeguards aren't there today.

Healthy caution, sure, but fear is a bad counsellor. Asking whether AI is safe is like asking whether a knife is safe. It depends on how we handle it and what precautions we take.

That's just real systems engineering, and it doesn't pay. 2010s SaaS and social product managers have infested the AI labs and turned everything into a product. I'd feel safer if Airbus were at the frontier instead.

When has society ever done precaution? Fossil fuels, social media, plastic, industrial ag. Post-WW2 America has always been full Leroy Jenkins, so reckless AI is on brand.

Imagine every big pharma CEO saying they can't be sure the new drug won't kill humanity, that they have no medical degree, and that there's a 10% chance of permanent catastrophe, plus a few lab escapes already. That's extraordinary enough on its own.

Or maybe Coxon actually considered the object-level arguments, which is deeper than OP's "robots are pretty hard to make right now" take.

So the rebuttal is that robots aren't good enough yet? What about when they are, and what about everything you can do entirely remotely? I think 10% is overstated, but saying the risk isn't worth thinking about is overconfident the other way.

His point is that anxiety isn't evidence. This fear has been around thirty-plus years, the people pushing it weren't right before, and they don't have new evidence now.

The 10% number leans on recursive self-improvement and an intelligence explosion. Would he call a 12% chance of a hard takeoff in 8 years absurd too? He never touched any of that literature.

Hitler didn't need robots. He used humans. Why change what already works?

Biology has been trying to grey-goo the world for billions of years and the world isn't that trivial. This is magical thinking from CS people who think saying the right words to a machine is enough to impose your will. Extinction needs an execution phase.

The guy opens by mocking his college self for sneering at people who don't understand tech, then concludes we shouldn't expect the public to understand LLMs, bioweapons, or extinction biology. Nothing changed in a few decades.

Sober and informed, we need more of this. Demand evidence for extraordinary claims. But "you have nothing to fear" is also an extraordinary claim, and AI risk, however small, is real.

Is saying we don't need to fear human extinction in the next ten or even hundred years really an extraordinary claim?

The burden of proof sits with whoever makes the extraordinary claim. If someone says you're an extinction risk to humanity, do we lock you in a bunker until proven otherwise?