MIT CSAIL: GPT-6 Astra reads raw weights, edits 0.016% of Qwen3.5-0.8B to lift MATH-500 from 7.9% to 48.1%
- MIT CSAIL authors Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab report that GPT-6 Astra can tell what a small network was trained to do, including which game, just by reading its raw weight matrices with no inputs or outputs shown.
- Given no environment feedback at all, Astra writes small game-playing networks from scratch that beat networks trained with PPO for 100 million steps, and it is never allowed to write a training loop.
- On Qwen3.5-0.8B it finds the weights behind a reasoning loop that Qwen's own model card warns about and edits 0.016% of them, raising MATH-500 from 7.9% to 48.1% and tripling IFEval under the same sampling settings.
- The authors call the method Weight-Space Programming and argue LLMs will soon optimize neural network weights natively by reasoning about them, replacing hand-designed symbolic optimizers like Adam or Muon.
- The claim covers reading and rewriting existing computation rather than teaching new knowledge: Astra locates where a model loops or routes badly and adjusts the weights rather than inserting facts it did not have.
Hacker News opinions
This is massive and I honestly did not see it coming. A model that looks at raw weights and names the game the net was trained on? How do you even train that?
It feels completely obvious in hindsight, which is its own warning sign about how I think about this stuff.
Slow down though. This is a multi-trillion parameter model fixing an 800M parameter one. Nothing here says it can improve something its own size, let alone itself.
The Qwen math thing is being oversold. Qwen already knew math, it was just looping, 85 of its first 96 answers hit the token limit. Astra found the loop and stopped it. They tried the same on Gemma 3 270M, which doesn't loop, and got nothing.
That matches what I read. It's killing a direction in activation space, basically the Arditi et al. 2024 refusal-direction result. It reweights and reroutes existing knowledge, it doesn't insert new knowledge.
Worth adding that the Qwen example used 50 batched iterations, so it could check its edits against downstream tasks. Not one blind shot.
So backprop is not the only game in town anymore. That's the part I care about.
I asked Astra whether it could do this and it said no. Make of that what you will.
My guess is attention does something like PCA over the pasted weights conditioned on the prompt, then writes back the transformed version. If that's right, small models won't be fixing big ones this way.
Labs are already training on ML papers and interpretability work, so this may be more automation than alien. Still looks like a big step for research itself.
If this holds up it belongs next to Attention Is All You Need. Editing a brain to make it better at math is not something humans can do.
The self-improvement leap is the scary part. Everyone expected AGI to rewrite its own code, not hand-edit weights, but here we are.