Fable 5 reaches a 2,726 NanoGPT speedrun record, closing 81.7% of the human gap

I want to know how much run-to-run variation each model has. The graph only shows the best validated result per model.

I still do not understand what an optimizer run is or why it measures research ability. The post should explain that instead of sending readers to Anthropic's evaluation.

My reading is that each model gets eight attempts to reduce nanoGPT below 3.28 loss under time and token limits. But 18 models times eight runs is 144, not the claimed 153 runs.

They are asking models to improve a small model's training, repeatedly test changes, and use results to choose the next change. Different seeds repeat the autonomous session to measure variance.

I do not see why Fable 5 ran at high effort while Opus 5 ran at max. Different effort settings make this look like an experimental error, even if models can now change effort dynamically.

The claim that good traces preserve weak signals makes me wonder whether a harness with a better history log would change the result. Small prompt changes such as asking the model to track weak signals or consider novel ideas might also matter.

I am watching the still-running Grok 4.6 run. I think it is a good fit for this benchmark and plan to check again in a few days.

I co-authored METR's NanoGPT speed-run post, and I worry about contamination if these models start from the original baseline. The token-scaling plots also show some models have not plateaued, so this is a performance-at-cost bound, not a full capability ceiling.

I am confused by GPT-5.6 Sol because its note says it spent much of the run waiting. That may stretch its time-axis result, and notes about older serial versions of program.md mean the comparisons are not fully apples-to-apples.

Grok's result looks poor to me. I cannot tell how much is model weakness versus a bad harness, but building a better harness is much cheaper than building a model.

Models that have spent long periods in multi-day LLM training environments may have an advantage on long-horizon benchmarks like this. Others may simply lack that experience.

GPT-5.6 Luna looks unusually good for a cheap model. It is near Sonnet-level performance here.