Berkeley study: coding agent harness choice barely moves success rate but swings cost up to 5x

Does this extend to open models like GLM 5.3? If so, just swapping the harness to Pi halves my cost?

Using Pi with local models feels way snappier than OpenCode or Claude Code for me.

Provider middleware matters too. I benchmarked my own harness against Kilo Code and on together.ai the model kept botching edits, looping on tool calls or falling back to scripts, while on deepinfra there was basically no gap between harnesses.

Most of the extra weight in Claude Code and Codex is there for security and alignment. Whether it works is an open question, but leaving that out and calling it a tax is disingenuous, you're just turning insecurity into a negative externality. Pi skips guardrails and sandboxing entirely, so of course it's cheaper.

Sandboxing Pi isn't hard and adds nothing meaningful to the system prompt, which is what this is actually measuring. Wrapping it in nono.sh costs about zero tokens. Pi should still ship with some built-in sandboxing though.

Do you have a source for that? Claude Code's system prompt has a few safety lines, but the bulk of the bloat is tool descriptions, not alignment. And you can't sidestep Claude's guardrails by editing the system prompt anyway, you can already replace it with --system-prompt-file.

"Harness" here is really being used to mean "agent", which is worrying. Execution pattern matters a lot, parallel versus sequential, delegating to subagents, using more than one model. Optimal harnesses do all that and cost and performance shift hugely, so a single-harness analysis is shallow.

Subagents can stack turtles forever in theory, but there's a ceiling, because eventually a person or a legal entity has to own the outcome beyond git blame.

Honestly Claude Code feels mildly shitty to use in the way every half-finished vibe-coded project does. Not terrible, but ridiculous for a two trillion dollar company's flagship product.

As the model gets smarter you need to tell it less.

It goes both ways. Small models need a minimal harness like Pi or the giant system message and tool descriptions just overwhelm them.

My own testing lines up with this. Having a coding harness matters, but the differences between them are overstated. I swapped OpenCode for a thin wrapper around Pydantic-AI, mostly to compartmentalize access control, and keeping the surface tiny means I still understand and can adapt the whole thing.

What keeps me on Claude Code is auto mode with its classifier and the soft_deny/hard_deny settings. When I'm debugging Kubernetes workloads I don't want the model deleting a pod. Codex is stuck on static allow/deny regex rules, and the model can still find an unanticipated way around them.

Claude Code and Codex charge users for massively bloated one-size-fits-all system prompts, safety instructions and stuff users don't want. If you're paying API rates for Claude or OpenAI models, almost every other harness beats them on cost.

If I use Claude Code or Codex but route requests through OpenRouter, do I lose the caching and other provider-side benefits? Or does it only matter if the request goes straight from harness to provider API?

Caching is handled at the inference layer by all the major providers, so it works as long as you're hitting the same model on the same provider.

If the gap is mostly system prompt size, the tax should shrink on longer-horizon tasks.

We need better harness benchmarks. There's no reliable source testing the main harnesses against all the open source models, and I wish Pi discussions didn't always use tokens as the metric. Pi is token efficient, but how does it compare to OpenCode if you don't care about token count? In my experience the harness is mostly polish against failed tool calls and bad edits, it doesn't change the underlying intelligence.