Small transformer reaches 44% on ARC-AGI-1 for 67 cents after 1.5-hour RTX 5090 training
- The author reports 44% on ARC-AGI-1 public evaluation after training a small autoregressive transformer from scratch for 1.5 hours on an RTX 5090, at a stated cost of 67 cents; it also scores 7% on ARC-AGI-2.
- The method tokenizes each ARC input-output pair and trains at test time on puzzle training examples, then uses color and dihedral augmentations at inference and submits the two most common inverse-augmented outputs via AAIVR.
- The update replaces GELU and LayerNorm with SwiGLU and RMSNorm, expands the model from 4 to 8 layers, switches AdamW to NorMuon, and reduces augmentation count while using FlashAttention and FlexAttention kernels.
- The loss now covers only output tokens rather than input tokens, which the author calls supervised training; reported ARC-AGI-1 performance rose from about 40% to 44% despite worse test loss.
- The training data includes 347 non-overlapping ARC-AGI-2 tasks after filtering the 773 ARC-AGI-1 puzzles repeated in ARC-AGI-2; removing those extra tasks reportedly retains about 40% but needs roughly twice the compute.
Hacker News opinions
I do not see much value in optimizing for one benchmark. A system that only solves ARC puzzles is not a general LLM or coding model, and a cheaper non-LLM method might do better if that is the only goal.
I read this as a response to the LLM hype cycle. It shows that a small, purpose-built ML system can tackle these problems without treating LLMs as the answer to every task.
The 67-cent figure is easy to misread. Spending $100 will not scale results linearly because performance hits a ceiling and extra compute has diminishing returns.
I think the low cost is the point. People assume LLMs are the universal solution, but specialized methods can solve hard, bounded problems for very little money.
That criticism seems unfair because performance versus compute often saturates. Reaching the plateau sooner is useful, and more compute can still improve the peak after other scaling problems are addressed.
I wonder whether general reasoning exists at all. Humans may look general only because we have encountered a huge range of stimuli, while still relying on specialization.
I wrote this to test sample efficiency with a small autoregressive transformer trained from scratch, not an LLM. Earlier ARC results mostly relied on costly LLM scaling, fine-tuning, complex architectures, or far more training compute.
I am curious whether training longer kept improving performance or whether it plateaued. Would spending $67 rather than 67 cents produce a major gain?
Most modern LLMs are autoregressive transformers internally, so calling this non-LLM seems mostly about the scale and training setup rather than the architecture family.
How does this perform on ARC-AGI-3? Schema Harness reportedly reached about 99% on ARC-AGI-3 Public with Claude Opus 4.8 and Fable 5, so I would like to know whether its ideas can improve mdlARC.
It will not do well on ARC-3 without substantial changes and work. I also think the architecture comparison in that video is misleading because it did not use proper baselines.
This made me want to spend time on non-LLM work again.
I have been thinking about ARC-AGI-1 in similar terms, and this framing puts the idea clearly.
I saw criticism that using inputs during pretraining is leakage. The prior blog had a community note making that claim, so I want to know how the author answers it.