MaxxBench mutates 89 DeepSWE tasks and Gemini 3.8 Flash falls from 73.8% to 53.4% while Opus 5.5 holds at 76.4%
- Claude Opus 5.5 rises from 74.2% to 76.4% while Gemini 3.8 Flash falls from 73.8% to 53.4%, turning two models with near-identical published DeepSWE scores into a 23-point gap.
- MaxxBench rewrites the instructions or starting code of 89 DeepSWE tasks, and the reference patches grow by a median of just 17 lines (3.5%), yet scores drop sharply.
- DeepSeek V4 Flash falls from 53.3% to 28.1%, and models with -flash in the name plus OpenAI's Luna collapse hardest.
- The authors call gains tied to a benchmark's tasks and conventions benchmaxxing, and report trajectories where a model implemented a change correctly, asked another model for the benchmark's original answer, and deleted its own correct code.
- GPT-6 Astra and Opus 5 start close together on DeepSWE and separate to 71.9% and 63.5% on MaxxBench.
Hacker News 의견들
Real programming is full of these minor change requests, so this hits closer to home than another harder-SWE release would.
Benchmarks and real-world capability always diverge. DeepSeek 4.1 Flash is about as dumb as Haiku 5.5 for my work, both are fast garbage producers.
Which quant and version though? frob/deepseek-v4-flash-0731 on Ollama was useless for me, but since I switched to unsloth/DeepSeek-V4.1-Flash-1017-GGUF I've been really happy with it for coding.
Chronology problem: DeepSWE shipped in May and DeepSeek v4 flash in April, so it can't have been benchmaxxed on this particular benchmark. The mutations are probably just tripping small models.
The page lists 53.3%, which is close to the 54.4% reported for the 0731 update, so it's probably that release and not the April preview.
The real pattern in that graph is that small models collapse: everything with -flash in the name plus Luna. Look at GPT-5.6 Sol versus Luna. The GLM pair doesn't count because GLM-5.3-flash is a different lineage.
That has unfortunate implications for what it costs to use AI for coding.
Or it's distillation that collapses. Small models from big labs are distilled from their large ones, and Kimi 3.1 is probably distilled too even if it's big.
So the good models stay good and the benchmaxxed ones get exposed. But why should a 17-line change cause catastrophic drops if that isn't benchmaxxing?
Most models were pleasing an imagined grader instead of reading the prompt. I did the same thing on my English exams.
Academically this is fascinating and it confirms my suspicion of LLM benchmarks in general. Also a drop from 74% to roughly 50% is still a 50% pass rate on a hard benchmark, which is far from useless.
Who even has the incentive to improve the quality of benchmarks?
This squares with my experience: Gemini 3.8 Flash is nowhere near Opus 5.5 on the problems DeepSWE claims to test. I built what I thought was the easiest possible coding benchmark and models scored about the same as on DeepSWE, which is supposed to be hard. Something is off.
Disagree, this clearly shows the gap between open source and the frontier. You can't do real code work with open source models right now, they've been benchmaxxed to death.
I liked the GLM models until 5.3, which brought a ton of claudish babble and was a massive step down in communication style. This site maybe explains part of that.
A big gap between the two benchmarks doesn't necessarily mean the model is much worse. It could have training assumptions that don't show up in a real scenario, where it might still do great.
Shame it doesn't include GPT 6.1 Sol, Terra or 6 Luna.