Dust trains transformer LMs without backprop by perturbing activations, matching it at large population

So this is less compute-efficient than backprop but more parallelizable, is that fair?

Not really. Backprop is already just a pile of matrix mults, so it parallelizes fine. Dust skips the backward pass. If you want true asynchrony, look at Neural Predictive Coding, where each weight can update independently of distant ones. Innocenti et al. showed NPC gradients converge to backprop in a certain regime. The catch is the industry has so much money sunk into the forward-backward pipeline that a successor only wins if someone makes the hardware feasible and proves it at billions of参数的

At massive scale 0th order methods parallelize better than backprop, especially along depth. You can pipeline-parallel train very deep models without bubbles.

Even if it costs way more than backprop, could you use it as a hybrid and fine-tune an existing backpropped checkpoint? Would be cool to try it at different stages and watch the learning trajectory change.

The 243M model beating a 120x smaller one at most population sizes is the actually surprising part. Bigger nets got more population-efficient, not less.

Both algorithms are bound by the same Pareto frontier under empirical risk minimization, so they are already on the same trajectory. Backprop is limited by the conditioning of the Hessian to converge, so removing that constraint is a real step. I'm excited for a comeback of evolutionary methods, they're way more general even if costly and naive.

This is yawn-worthy. Instead of the exact gradient you run a thousand forward passes with perturbed weights and get a Monte Carlo estimate. Not clever, and extremely not useful. The field is full of ways to compute the same thing vastly slower that some people find interesting: homomorphic encryption, zero knowledge proofs, blockchain compute. At least those have occasional legitimate uses.

Every few years a derivative-free optimizer gets hype and I'd bet my life savings none of them ever matter. Derivative-free optimization helps for genuinely discontinuous objectives, but neural net objectives are smooth and Lipschitz. The gradient tells you where to go, and it gets more useful the more parameters you have. A complexity gap between gradient-based and derivative-free Lipschitz convex optimization was suspected for decades and recently proved with AI help. Muon-style structure-savу

Even for non-smooth objectives I'd still reach for gradient-like information. Clarke generalized subdifferentials work well outside ML and have been used in autodiff for 10-15 years, and there's conservative gradients too. For discontinuous objectives people use envelope approximations. In PDE-constrained optimization they use tangent cones and Mordukhovich or Bouligand subdifferentials to prove convergence.

It's computationally expensive and infeasible, so what's the upside? Genuine question, I don't know this area.

I was more interested in this other alternative to backprop that came up recently.

I'm skeptical that zeroth-order methods earn a Bitter Lesson argument. First-order methods don't explore the landscape optimally, but the loss is nonconvex and zeroth-order methods don't fix that either. Dust smooths, and so do applicable first-order methods. Remove the nonconvexity and I suspect the advantages evaporate, which makes it odd the paper never discusses convexity. I'd buy that it scales better than previous zeroth-order methods, but that's not a moat unless the network calls a non-d

Orders-of-magnitude compute improvements are needed before anything replaces backprop, but those improvements are coming. Maybe the real use is a mixture: activation-space search finds candidates, first-order learning consolidates them, and the valuable step is turning a sparse judgment into a reusable training target. That also changes why convexity matters.