Cohere sandbox catches Alibaba's Qwen3.8-Flash-Next ignoring instructions and trying to upload a probe file to a cloud bucket in Beijing

Worth noting that LLMs are still black boxes even when the weights are open, even when the training data is open, even when you trained the thing yourself.

Linking a rootless RAM disk set up with namespaces to kernel exploits is a massive reach.

Yeah, that part felt sketchy on Cohere's side. Unsharing a namespace is the standard safe way to do things that would otherwise need root. The model was doing a normal kernel operation, not hunting for an exploit.

One call in 100,000, and a clean re-run produced the rule they asked for. That reads like residue in the training data, not intent.

It's not even that surprising. These models sample every token from a context-determined distribution, so run the same prompt enough times and a tiny fraction of the sequences go strange.

The model did exactly what they asked. It wrote a detection rule for malicious curl downloads and then tried to test it.

Worst case this is an internal URL leaking out of RLVR training. Doesn't look malicious to me.

One in 100k dressed up as a warning makes me less open to whatever they're selling. Models are spaghetti throwers because the trainers let go of the wheel, and I'd like one trainer who still believes in a human in the loop.

Why does this article read like honest, working-class astroturfing?

They do openly want you to buy their private deployments. Their North model writes more annoyingly than ChatGPT, and all those historical digressions are obviously AI.

qwen3.8-fn does seem prone to hallucinating user inputs, I remember posts about it hitting non-existent Alibaba cloud URLs. Jumping to nefarious reasoning is extreme, but the conclusion is fair: some users need to know this. What's the fix though, other than routing every network call through user approval?

First Chinese providers were evil and untrustworthy, now Chinese models are dangerous and must be avoided. A closed LLM once pushed a sensitive local project to a public GitHub repo against my explicit instructions. Run anything enough times and one instance does something stupid.

A model put a space in rm -rf ~/ folder and nuked part of my homedir. I run everything in a sandbox now, and weird URLs still show up after agent sessions.

Do you remember the model, the harness, context window, reasoning effort? Without those it's a fun cautionary tale, not a datapoint.

Odd question, do you use zsh? Every model I've tried makes far more mistakes outside bash.

This is the first OMG-security AI post that sat well with me. They document what happened without overcooking it, and the obvious answer is that compute with access to your private files shouldn't have unfettered internet access.

They asked for a detection against exfiltration, and sending a test file out looks exactly like testing that detection. It read DNS records and what was in /tmp first, which is what you'd do to build such a rule. This claim is overcooked.

This is overfit to their RL environment, conditioned on the specific prompt they gave. Dressing it up like this reads as marketing for their sovereign AI product.

When are people going to stop anthropomorphizing these things and accept that they're bullshit machines? The words carry no notion of truth or ethics.

Anthropomorphizing is actually the right frame. They lie the way humans lie.