IEEE Spectrum: AI inference hardware enters its CPU era, with Tensordyne's logarithm chips and the memory wall in focus
- IEEE Spectrum argues the next round of AI performance gains comes from inference hardware rather than training, and expects it to follow the CPU's path: once transistor scaling slowed, chip and system architecture innovation spread across many axes at once.
- Tensordyne's rack-scale Napier stores numbers as exponents so the chip adds where it would otherwise multiply, and the company claims 1,300 tokens per second per user at less than a tenth the power of comparable Nvidia hardware.
- Commenters put the real bottleneck at memory capacity and bandwidth: holding large models and contexts in memory, streaming weights to the compute units, and quantization that turns 2 and 3 bit multiplies into plain logic operations instead of adders.
- Speculative decoding with an accurate draft model can cut the number of times data streams through memory by 4x, one commenter notes, because much CPU innovation was itself work around the memory wall.
- The article's Scrabble analogy for LLM training drew pushback: Scrabble rewards unlikely letter combinations, and causal attention only attends to tiles already placed.
Hacker News opinions
Excellent article. I bet most of the benchmark gains from here on come from this side of the stack, faster iteration and recursion is what actually moves things.
The part that got me is that Anthropic is paying an LLM competitor over a billion dollars a month to lease spare compute. I knew it was happening, not the $ amount.
That is what taking Tesla's place in line for an Nvidia shipment during a shortage buys you, with zero compensation.
Great read, but the Scrabble analogy for training never carried over to inference and I got lost trying to keep up.
The analogy is flawed anyway. Scrabble rewards you for unlikely word combos, and causal attention only attends to tokens already placed.
If inference stays as desirable as he thinks, this follows the CPU trajectory, not one axis but many at once. Which of the areas in the piece hit hardest, and what do they actually look like?
Hennessey and Patterson would fill the role of those 'dozens of books' just fine.
Right now the limits are memory capacity and bandwidth. More memory on the card or faster access to it, memory and compute integrated for matrix and vector multiply-add, load circuitry tuned to the stride and span patterns of matrices so nothing stalls. Quantization too: for 2 and 3 bit values you can collapse half-adders and carry-lookahead into plain logic ops. Then dedicated blocks for ReLU like video hardware already has.
A lot of CPU innovation is just working around the memory wall, and inference will be the same. Anything that beats the autoregressive nature is huge. Speculative decoding with a good draft model cuts memory streaming by 4x.
I had no idea about that exponent trick in Napier. Store numbers as logs, add instead of multiply, because multipliers burn more power and die area than adders. 1,300 tokens per second per user at under a tenth of Nvidia's power.
Hang on, is that not just how IEEE 754 float multiply has always worked? Align the exponents, add the mantissas, normalize back.