Goodfire essay argues neural networks are already clean and mostly interpretable
- Nick Cammarata of Goodfire argues that polysemanticity in early models came mostly from ordinary, fixable training issues rather than superposition.
- With architectural improvements and no change to model size or training data, 99-100% of neurons in InceptionV1's last layer are now easily interpretable.
- The Zoom In circuit once presented as superposition, where a car feature spread into unrelated dog neurons, is reinterpreted as ordinary generalization and composition.
- The author says the model stays broadly legible across a 300× range in size, with smaller models using coarser concepts and larger ones splitting them into specific cars.
- The essay claims modern neural networks, including models up to trillions of parameters, can be studied directly instead of first untangling them with decomposition tools like sparse autoencoders.
Hacker News opinions
I read clean as each neuron mapping to roughly one nameable concept. Nick's claim is that the old puppies-and-murder style mixing was mostly an ordinary training failure, so it's fixable.
Architecture matters a lot here. A trillion-parameter transformer with many layers is way easier to read than a few-million-parameter net evolved over one giant layer. The redundancy helps humans lock onto patterns.
Feels like the distill.pub days. Zoom In was the reference point for years, so seeing that car-to-dog circuit reinterpreted is wild.
I'm not ready to say superposition is wrong. Training SAEs or transcoders adds another layer of abstraction, and I like studying the neurons that already exist. Still, I'd keep an open mind.
The laggy 3D city at the top is useless and killed my phone. Block JS and it's fine, but a plain GIF would do the job.
The 'simple thing at absurd scale' line is doing a lot of work. If it holds up it's huge, but so far I've only seen a blog post, not a paper.
A glossary won't fix the readability problem for laypeople. I have a uni degree and still feel lost, and I don't even know what problem he stopped worrying about.
Understanding algorithms past a certain complexity is undecidable in general, so I'm skeptical this scales to 'everything is legible.'