Magic claims its pretraining recipe matches DeepSeek V4 Pro Base with about 50x fewer FLOPs
- Magic says its pretraining recipe matches DeepSeek V4 Pro Base on its heldout loss tests using about 50x fewer FLOPs, which it estimates as roughly half of GPT-3's pretraining compute or about $0.5M on GB200 hardware.
- After scaling its run 10x to an estimated $4M, Magic reports lower bits-per-byte loss than publicly available open base models on private code repositories, heldout research papers, and generated reasoning traces for private math problems.
- Magic measures bits per byte to normalize tokenizer differences, fits scaling laws against 6·N·D training FLOPs, and projects that DeepSeek V4 Pro's recipe would require more than $100M to reach its reported capability level.
- The evaluation uses Magic's codebase and acquired private startup repositories, recent low-citation papers, and private math problems; Magic says it removed documents matching training data through 96-character normalized-text windows or Jaccard similarity thresholds.
- Magic checked open-model log probabilities in vLLM and SGLang on GB200 and GB300, found backend issues, and partnered with Fireworks to verify baseline log probabilities in its inference engine.
Hacker News opinions
If the claim holds up, matching DeepSeek V4 Pro Base with roughly 50x fewer FLOPs is a very big deal.
I want a paper explaining the method. Nous's token superposition work improved pretraining FLOPs, but nowhere near 50x, and I wonder whether tokenization is part of Magic's result.
Releasing the method could save a huge amount of power, although Jevons paradox could raise total consumption if lower costs drive much more training.
Training improvements are easy to copy. If this is real, a major lab may buy Magic, or Chinese labs may rediscover and open-source the trick quickly.
Bits per byte is next-token surprise normalized by text length. Lower is better because the model assigns more probability to the actual text.
I expect costs to keep falling, but frontier labs will keep pushing the boundary. A model that is 10x cheaper is not enough for the hardest tasks where the economic gains are.
I'm concerned they may be comparing their pre-trained model against post-trained models. BPB is optimized in pretraining, while post-training often hurts it, so that would favor Magic unless the baselines are truly base models.
They say the comparison is against base models, so it seems fair as written.
This is what pretraining teams at labs work on. Compute efficiency is the metric because they assume scaling produces intelligence.
A 10x pretraining reduction does not necessarily mean lower energy use. A lab can train larger models in the same time, or run far more experiments and shorten the feedback loop by 10x.
The charts may mostly show a data mix tuned for code and science. The model could still have weak general world knowledge or subjective-preference performance, and the knowledge tests focus on domains where scientific-writing patterns help.
Magic has made large claims before, including its 2024 100M-token context post, without visible products or published research. I want evidence beyond a company blog.