Real-SWE puts coding agents on licensed private enterprise codebases, with Fable 5.1 leading at 38.8%

I want to know which benchmarks matter now. Terminal-Bench 4 seems useful because it is recent and harder to benchmark-optimize, though it measures sysadmin and DevOps work more than programming.

Since 5.3-codex with extra-high thinking, I have offloaded most coding and other tasks to Codex. Astra lets me hand over whole projects without watching every individual task.

I do not see the value in a benchmark that cannot disclose the code or much about the companies. It sounds like a pinky-promise result.

I think private benchmarks can still gain value if the evaluator builds a record of reliability. You trade transparency for a test that is harder to game.

Private tasks may be necessary to prevent benchmark saturation. If every model uses the same tools on the same tasks, the comparison is still useful at that point in time.

I doubt serious companies hand proprietary crown-jewel code to a third party, licensed or not. I would like to know who supplied these codebases.

I suspect many of the codebases are abandoned small or medium startup products bought by the line of code. There may be fake codebases too.

I do not find this result very useful in practice. Astra is about 5 times faster for me than Fable and much less annoying to use, and I can try the few viable options myself.

The benchmark is measuring resolution rate, not speed or how pleasant a tool is to work with.

Gemini 3.8 Flash ranking this highly makes me distrust the benchmark. In my real work it loops over the same files and makes hundreds of tool calls for a simple change.

I use Gemini 3.8 Flash in Antigravity and prefer it to many Pro models. With a good harness, tests, and review interrupts, it made progress on compiler-like work that older models including Opus missed, and solved a 10,000 LOC problem in under six hours.

The harness matters more than a single model label. Gemini CLI was deprecated on May 19, 2026, and Gemini 3.8 Flash uses agy, or Antigravity, now.

I would test these private codebases for model contamination repeatedly. From running botsbench.com, I learned that supposedly private material may already be in model training data.

My experience matches the Fable 5.1 lead for large feature work. Fable assesses priorities better, while Astra often completes 90% of a task but misses the part that matters and then spends hours on irrelevant edge cases.