Dan Luu tests 26 prompts and four skills for agentic Rust verification

I think we keep throwing benchmarks at models while overlooking the harness around them. Harnesses need tuning too, and changes can improve one tested function while losing another.

I read this as manual mutation testing, not a mutation-testing framework. Why not fail the build when too many mutations survive? An agent can ignore a TDD instruction, but it cannot ignore a build gate.

Would an automated mutation gate really solve it, or would agents just learn a new way to game the metric?

I suspect the agent saw "use mutation testing" and interpreted it as manual mutation testing, or lacked an installed tool. I'm surprised the article did not call out that distinction.

I use automated mutation testing as a zero-missed gate, not a percentage threshold. A test recently passed on both fixed and mutated code, and only the gate caught it. One surviving mutation can disappear under a percentage threshold.

I think the experiment decouples testing from architecture too much. Most effective testing comes from code architecture, and I get better results by requiring DI or hexagonal architecture plus a trivial coverage check.

I read it differently: the same agent writes the code after receiving the test prompt, so it can choose an architecture that fits the testing strategy.

I prefer tests of behavior over structure. Some coupling to implementation details is unavoidable, but I would rather substitute a fake in a centralized composition root than monkey-patch a dependency separately in every test.

The formal-methods results seem plausible because there is far less Verus, TLA+, Creusot, and Lean code than ordinary code. I would rather see a manually written high-level specification for a nontrivial system, then test whether an agent can refine it into an implementation.

For formal methods, I want to write partial specifications that tell the LLM what I intend, then have proof work check that intent. Throwing tools at an LLM without a specification is predictably weak.

My problem with agents is that they run huge test suites for tiny changes. I have to put rules in agent memory and hooks in the repo: form a hypothesis, spot-check first, then widen testing only when needed.

I thought agents tested reasonably well until I inspected their tests. I found assertions such as assert(CONSTANT_CONFIG == valueOfConfig) and checks that a prompt contains a repo URL.

I take the results to mean that testing prompts vary widely in error rate, with the worst around twice the error rate of the best. The default prompt seems fairly good, and many custom prompts are worse.

I have seen agents think of more unit-test edge cases than people do, but skills can make them mechanical. With Superpowers, the agent follows TDD by testing whether a Send button exists, yet misses business behavior such as allowed sending, failures, and duplicate submissions.