Google Research's sleep framework lets a deployed LLM consolidate new facts into its weights during offline phases, claiming 10M-token retention
- Google Research added a sleep phase to continual learning: a deployed LLM alternates between an active wake phase that stores incoming context in short-term memory and an offline phase that consolidates it into parameters, which the authors say avoids catastrophic forgetting and fixed knowledge cutoffs.
- Consolidation works through parameter expansion, opening new low-rank experts inside pre-allocated Mixture-of-Experts blocks so new knowledge lands without overwriting older weights, then transferring short-term memories upward with generalized knowledge distillation and RL-based imitation scored by Levenshtein distance.
- After consolidation, synaptic pruning resets the high-frequency temporary parameters to free capacity for the next active session, while the dreaming stage has the model self-generate practice examples and refine them with reinforcement learning to sharpen reasoning.
- The framework runs inside a fixed model architecture under human-defined training rules, and the team reports retaining long-context knowledge up to 10M tokens; the paper is arxiv.org/pdf/2606.03979 by Ali Behrouz and Vahab Mirrokni.
Hacker News opinions
This is a surprisingly well-written research post, Google does have some seriously good science communicators. So they're holding back unweighted parameters in the architecture, populating them with new context during sleep, then using the dream phase to reinforce what lands in those MoE weights?
Sort of. Short-term memories pile up in a buffer as you interact with the model, then during sleep they get scored and ranked and only the useful bits get written into the weights. The neat part is the update is a transform where the end state is the old weights plus the semantic knowledge, not just dumping whole memories in.
Weird to see this built off the KV-cache in their diagrams, completely different from Databricks RAFT. My team built Dreamtime earlier this year pulling from user nudges on bad responses instead of the token layer, sawtooth-style: accumulate, flag the dissonance, compress, retire the old one. The real killer for customers is audit, you need to answer when a change happened and who made it, and that's hard if you can't keep lineage back to the original session.
You forgot the reference [2]...
Feels like one of the missing pieces on the road to AGI. Add critical thinking to the cleanup pass and we're basically there.
How is that good? That sounds like a bleak future to me.
If there are any jobs left that pay humans, this eats a big chunk of them.
Reminds me of DreamCoder, one of my favorite ML papers.
The actual paper is arxiv.org/pdf/2606.03979 if anyone wants to read it.
Does this shit actually work lol
They ran this on tiny models, Llama 3B and 8B, which are easy to improve because they aren't good to begin with. Moving the weights of a multi-trillion-parameter model would take way more tokens, and I'd want to see it against an RLHF control group, not just in-context learning and TTT.
I'm seeing way more typos online than a few years ago. Half convinced some of it is deliberate so readers know a human wrote it, and the typos in this preprint definitely read human.
Parameter expansion is the limiting factor here, it all depends on how efficient that part is.
For the neuroscience side, there's the KIBRA and PKMzeta memory glue discovery, work showing the brain stores multiple copies of each memory, and evidence memories stick around after half the synapses are gone.
Who can afford to sleep in this race to the bottom?
Love that they tested it on Qwen 3. I want to know if they tried putting the new parameters in system RAM or on an SSD, shame they didn't publish that part.