Coding-agent harness study ablates 176 settings across four models: context management and bash-only tooling move cost more than accuracy

Only skimmed the PDF, it's long. The car analogy is right: if car A beats car B it isn't necessarily the engine, could be tires, gearbox, weight. Same model, and you can wire it as a ReAct loop, plan-and-execute, hybrid, stateless, RAG-based, depending on whether you want accuracy or depth.

The conclusions all read as duh to me. Bash-only tooling beats predefined tools for bash-capable models? Yeah, no kidding. The last line about harness design being a lever I agree with regardless of how they got there.

Duh or not, this means we need a formal list of what each model is actually good at, and then pick or swap harnesses to fit. Like an LLM recipe book. That goes for local models too, where how you run them matters as much as the weights.

Testing the obvious things is valuable science. 'Everybody knows foul air causes sickness' was obvious too. These studies shrink the assumption space, which is the whole point of a solid foundation.

Same-lab model plus harness gives a much better experience than the general purpose harnesses, in my experience. Claude with Claude Code just feels tuned.

Not really. There was a recent benchmark with Claude and Codex showing no difference using a harness like pi agent versus their native ones, and pi was cheaper per task.

Also, the conclusion isn't 'bash-only is better for everyone.' It's bash-capable models plus bash-only tools beat bash-capable models plus predefined tools. Which basically says MCP was a crutch that helped until models learned bash, and now it's dead weight.

Todo and task-tracking tools like TaskCreate/TodoWrite are gone on Opus 4.8, Sonnet 5 and newer, you have to set CLAUDE_CODE_ENABLE_TODO_TOOLS=1 to get them back. Anthropic seems to agree frontier models don't need in-session planning tools. Issue 80487 on the claude-code repo.

This was run on Nemotron and Mistral models, so it's not relevant to what people actually use now, the cheap Chinese models and Claude/GPT. Big miss not including Qwen or DeepSeek.

The point of the study was how different harness approaches scale across model sizes. Which specific models they picked is basically irrelevant to that question.

I'm getting tired of this argument. Nobody ever gives a counter example, just 'models moved on.' It's an in-depth empirical study by people who know what they're doing, versus a random comment saying nope because the models aren't the latest. And LLMs aren't fundamentally different from each other, they've functioned similarly for a while now.

How do Nemotron and Mistral actually compare to the frontier in your experience? That's the part I'd want to see before dismissing it.

The paper keeps saying 'bash capable' and never defines it. How do I know if my model qualifies? Luna would surely be bash capable. At that point any tool beyond Bash needs rigorous testing to prove it adds value.

I think it just means there's some parser that consumes the token stream and spawns shell processes from the command strings it detects.

It's self-defined: can the model use bash well enough that it doesn't need special tools. The paper agrees with you, adding tools that duplicate what bash does doesn't improve performance. Same story as RAG, outside narrow domains you don't beat handing the agent a shell and grep.

Great to see principled harness studies. There still aren't many benchmarks where a complicated harness consistently beats a very simple agent. I'm biased, I wrote mini-swe-agent, which started as 100 lines total, and it's used in DeepSWE, terminalbench and programbench, still top of the TB3 ranking though it wasn't run with the best models on TB4.