A linear program reproduces Astra's 8-hour Qwen weight edit in about a minute, with 8x fewer changed weights
- The edit that lifted Qwen3.5-0.8B from 7.9% to 48.1% on MATH-500 is a rank-1 change along a single loop direction u, so two linear program solves of about a minute each match it with 8x fewer changed weights than Astra's roughly 8 hours and 28 forward passes.
- The edit targets thinking loops: 93% of the base model's MATH-500 answers hit the 32,768-token limit, and after the edit that drops to 33.1% while IFEval rises from 21.0% to 62.3% and GSM8K from 13.8% to 67.1%.
- Mechanically it subtracts alpha times (u transpose w_j) times u from the
down_projcolumns in layers 15 and 19; every column at alpha=1 gave 33%, alpha=3 gave 48%, top 64 columns at alpha=6 gave 55%, and a random direction gave 12.5%. - On Gemma 3 270M, which rarely loops, Astra could not meaningfully improve math ability: the authors describe weight editing as removing an obstacle hiding an ability, not adding one the model lacks.
- The HN thread notes the approach resembles refusal-direction suppression used to uncensor models, and several commenters argue an LP or SAT solver could express Astra's intent directly as constraints instead of trial and error.
Hacker News opinions
Nice analysis. The simplicity of this makes me think it should just be happening during training via regularization.
So is this a very powerful LLM like Astra editing smaller models? Could Astra edit itself?
No, it's a linear optimization algorithm from the 1940s editing a Qwen model with billions of parameters.
I don't get the downvotes on that. Weight editing is basically finding a semantic direction for the bad behavior and steering away from it, and with weight access I see no reason a model as big as Astra couldn't do it to itself, maybe even at runtime.
The first headlines made it sound like frontier models can read and manipulate weights directly, which seemed unlikely since they're trained on tokens and RL, not weights. This reads more like Astra ran small weight-editing experiments and found a good edit. Cool, and cool that the analysis finds a faster route, but a far cry from the coverage.
That feels like splitting hairs. Astra didn't edit the weights directly, it came up with a method that edits them. By that logic I don't save files because I don't flip the bits myself.
And we've just reinvented the refusal-direction suppression tricks people use to uncensor models. One step at a time.
You can't really train on second-order effects, and if you could you'd be stuck on third-order ones forever.
The real point is that an LP solver is a tool for expressing intent: kill this behavior, keep that one, as linear constraints. Astra's prompt was basically 'improve Qwen's math', so the solver cuts the trial and error and Astra could iterate on top of it.
I want an LLM that reaches for an LP or SAT solver or a prover without being prompted, not as JSON tool calls but natively in the inference engine. Hand it a word problem, let it state the formal structure, let the solver answer back in token space.
Good to see LP make a comeback. We've settled for suboptimal solutions because everyone is too attached to LLMs and wary of numerical analysis. Someone showed linear regression matching an expensive SVM a while back.
So 1940s operations research simplex is filling in a step of recursive self-improvement? Wild if true.
It wouldn't be RSI, it's just a different algorithm solving for the weights.
What's actually different from backprop here? Does the model reason about more parameters than a gradient step gives you?
Weight-space programming has no settled definition yet, but it means implementing weight changes with a trained model using forward passes, no derivatives needed.
Anyone doing low-dimensional weight-space manifold work? If you could shrink the dimensionality you could run global optimization and find sparse accurate networks, lottery ticket style.