Cornell guide traces how diffusion LLMs refine full text sequences in parallel
- Diffusion LLMs generate a full token sequence as an initial draft, then revise every position in parallel over several denoising rounds, unlike autoregressive models that emit one left-to-right token per step.
- The article identifies three autoregressive limits that diffusion generation changes: emitted tokens cannot be revised, generation takes one step per token, and causal attention cannot use future context.
- A masking diffusion language model is built by defining a forward corruption process for discrete tokens and training a reverse model to recover clean text from partially corrupted sequences.
- Modern diffusion LLMs use iterative refinement, post-training, and variable-length generation; the article cites Mercury 2, Google's Gemma Diffusion, and NVIDIA's Nemotron Diffusion as 2025 to 2026 releases.
- The underlying diffusion recipe comes from image models: a forward process progressively adds noise to training data, while a learned reverse process denoises it step by step.
Hacker News opinions
Good stuff, but it doesn't mention confidence. I'd recommend looking at DiffusionGemma and the other implementations too.
I've been studying these for a university project. Working through the ELBO derivation for a couple of hours was surprisingly illuminating once I understood the larger structures.
I wish more papers explicitly named importance sampling. Probability notation and the types of variables can be confusing when you're learning, though ChatGPT has been useful for filling specific gaps after I identify them.
I've been using Diffusion Gemma locally, and its GPU output tokens per second are very fast. The whitepaper says more time and compute could have improved it, but it's already quite usable.
For the image-diffusion background, the Welch Labs video on image generation is a good companion explanation.