Nature paper retrofits subword LLMs to run on raw bytes
- Researchers convert existing subword-based LLMs into byte-level models through a two-stage procedure called byteification, with minimal extra training.
- The resulting byte-level models outperform earlier byte-level approaches and excel on character-level reasoning tasks.
- The method reuses the ecosystem around the source model and reaches practical inference speeds by processing byte-level information efficiently.
- Subword tokenization obscures fine-grained information, which matters most for computer code and biological sequences where meaning depends on individual characters or bytes.
Hacker News opinions
Wow, Nature got rolled. Should have stuck to what they know. A lot of the applied AI work they were accepting was already hustling them.
What's bad about this paper?
Subword tokenizers were never causal in the first place, so BPE was peeking at future bytes the whole time. TIL.
Token-free LLMs have had a lot of research already. Byte Latent Transformers (BLT) and Charformer both go after this, so it's odd we're still stuck in token land.
Aren't diffusion models token-free?
Check H-Net too. That's another token-free approach, and it's in the ICLR 2026 proceedings.
It's about chunking, which human minds do too. Go finer granularity and you need more permutations and compute to match what existing systems hit. GPT-4's tokenizer had extended tokens for code structures and was well ahead for a while, so the vocabulary matters a lot.
Weird paper. Models have had byte tokens for ages. They can already read and write individual bytes, and they have multibyte tokens on top of that.
The goal is only byte tokens, not byte tokens alongside multibyte ones, so failures like counting the Rs in strawberry go away.