LittleLearner tests whether LLM scaling or RL can exceed a K-5 training boundary
- LittleLearner trains 0.6B, 1.3B, and 5B models from scratch on LittleCurriculum, an 88B-token FineWeb-Edu corpus filtered to U.S. Common Core K-5 material, with concepts and vocabulary above Grade 5 excluded.
- The project reports that scaling, SFT plus GRPO post-training, and in-context prompts improve capabilities taught by the curriculum but do not meaningfully improve performance beyond K-5, which it calls elicitation rather than acquisition.
- Each bounded model has an unfiltered control with the same architecture, token count, and training recipe, intended to isolate the effect of its pretraining corpus.
- GRPO math post-training on MathCAMPS raises in-scope K-5 accuracy but does not recover beyond-K-5 performance, including when the post-training data is out of scope.
- The authors release base, GRPO, and chat-tuned checkpoints and propose experiments on RL-driven discovery, continual learning of concepts such as negative numbers, and comparisons with child learning.
Hacker News opinions
I initially thought it knew quantum entanglement beyond fifth grade, but I had mistaken the unfiltered model's response for LittleLearner. Its actual answer described people tangled in ropes.
This resembles Talkie, an LLM trained only on vintage material. The constrained-corpus idea is not entirely new.
I would rather get my 8-year-old's answer, "I don't know." The model seems not to know that its knowledge is limited or that other people may know more.
The Rayleigh-scattering answer makes me skeptical of the "elementary curriculum" claim. It talks about UV and infrared, and parts of that explanation are wrong.
Kids' science books do explain Rayleigh scattering at that age. The difference is that this model has near-perfect recall of the material, not a child's personality or understanding.
The filtering is where I want scrutiny. The paper shows excluded examples, but I want random training-set samples checked by humans; the dataset release should make that possible.
My concern with LLMs is their inability to say "I don't know" or have a reason to refuse an idea. Endless agreement erodes trust over time.
Some top-tier models do push back. Opus has told me an approach was a bad idea and explained why, though it will still comply if I insist.
There is active work on injecting humility and defining knowledge boundaries in LLMs. Nouswise has a paper and product aimed at this.
This reminds me of a two-year-old: an answer for every question, delivered with certainty. I want to know when a model learns "I don't know."
Ask it "why?" repeatedly after an answer. It is built to produce answers, not explanations, more like a Magic 8 Ball.