Anthropic's 2021 framework rewrites small transformer circuits for mechanistic analysis
- Anthropic recasts transformer computation in a mathematically equivalent form intended to make the internal operations of small models easier to analyze.
- The paper restricts its initial analysis to transformers with two layers or fewer that contain only attention blocks, unlike GPT-3's 96-layer architecture with alternating attention and MLP blocks.
- The authors identify induction heads, specific attention heads that explain in-context learning in the small models they study.
- The work frames mechanistic interpretability as reverse engineering transformer computations into human-readable mechanisms, with the goal of explaining known safety failures and finding others earlier.
- The authors build on the Distill Circuits effort for vision models and present this work as an initial transformer-focused counterpart.
Hacker News opinions
I wondered how successful the Distill Circuits work on vision models actually was.
As far as I can tell, the whole Distill project went on hiatus. Their 2021 notice says as much.
I've tried reading this several times, but it is very long. Is the time investment worth it?
If this is close to your field or you are seriously curious, I think it is worth reading. I put it in the same class as word2vec.
Neel Nanda's walkthrough is a good companion: . It is still long because the material is detailed and complicated.
I'm surprised mechanistic interpretability gets so little public interest given the unfamiliar capabilities of LLMs. I expect this and later transformer-circuits.pub papers to become foundational references.
The paper changed how I picture attention: it rewrites the linear algebra around Q, K, and V into larger but equivalent matrices that are easier to inspect. It also made me see the residual stream as the model's continuous communication bus, with attention heads and feed-forward blocks writing into it.