ICML position paper argues chain-of-thought tokens are generated scratch space, not evidence of model thinking
- An ICML 2026 position paper argues that intermediate token generation, commonly called “reasoning” or “thinking” traces, should not be treated as a human-like account of an LLM's internal computation.
- The authors say anthropomorphizing intermediate tokens confuses how language models work, encourages questionable research, and can lead users to treat generated narration as an interpretable window into model thought.
- The paper distinguishes the model's token-by-token output from human reasoning: an apparent “aha” token only becomes part of the next forward pass as context, rather than proving an internal state change.
- The paper is a revised fork of its April 2025 preprint, submitted as version 4 on June 9, 2026; the authors say it overlaps with v1's background but changes both its focus and argument.
- Discussion participants proposed treating traces as scratch space or generated internal monologue, while others said early trace inspection can still expose missing context or bad assumptions during long tasks.
Hacker News opinions
I’ve always been bothered by traces that say “Wait, that’s wrong” and then repeat the exact mistake. An “aha” token has no internal state change behind it, just one more token in the next forward pass.
By itself, “aha” says nothing. But if the insight follows immediately, the word can still help me locate the useful part of the trace.
I think of these as film-noir internal monologues in a document the LLM is generating. Bad advice like “keep cheese on your pizza by using glue” is bad whether it appears as dialogue or internal narration.
They seem closer to scratch space than reasoning. The model stuffs its context with related tokens, and that may help attention retrieve the right material.
I would discourage anthropomorphizing LLMs and treat thinking traces as black boxes. Opus 5 often reaches excellent conclusions despite reasoning that looks highly questionable.
I do inspect the start of a long trace for missing context or bad assumptions. If a 20-minute task is headed wrong in its first minute, I can stop it and save tokens.
There is an engineering consequence beyond terminology: if intermediate tokens are not faithful computation, they are poor audit artifacts. I would record inputs, model version, configuration, tool observations, and outputs so a run can be replayed.
I dislike titles that tell me what to do, especially when the paper admits intermediate tokens have a loose correlation with correct solutions. My solutions work, and that matters.
This is explicitly a position paper. Its job is to argue what the ML community should do, and the paper summarizes earlier empirical work from the group.
I do not think casual anthropomorphism is necessarily a real problem. “Reasoning tokens” is shorter than “learned prompt augmentation tokens,” and saying MySQL “thinks” it should read another config directory does not mean I believe it thinks.
A lot of people are not in on that joke. The ELIZA effect and AI psychosis are real concerns, and heavy LLM interaction may affect people even when they start out mentally stable.