Solo Developer Trains 3.8B Model to 0.384 CORE for $998
- A solo developer trained a 3.848B-parameter decoder-only model on 65.3B tokens for 43 hours across 8 B200 GPUs, reporting 0.384 CORE at a cost of $998.
- The final model uses Llama-style RMSNorm, RoPE, GQA with 24 query and 8 KV heads, relu² MLPs, QK-norm, logit softcap, residual scalars, and ResFormer-style value embeddings; 14 value-embedding tables account for 721.2M parameters, or 19% of the total.
- After an 858M FineWeb-Edu run scored only 60.45% on PIQA, the author switched to ClimbMix, used Muon for matrix parameters and AdamW elsewhere, and replaced cosine decay with a trapezoidal learning-rate schedule.
- Training used FP8
torch._scaled_mmon all three GEMMs and padded the vocabulary from 50,257 to 50,304, which the author estimates raised throughput by 33%; reducing context from 2048 to 1024 roughly doubled batch size at fixed memory. - A 1024-context 3.8B run reached 0.338 CORE after 57.3B tokens in 35.9 hours for $820, while extending training at 2048 context reached 0.384 CORE after 65.3B tokens.
Hacker News opinions
I wonder what the minimum budget is to get a model to score well on AIME.
I keep seeing LLM-written prose everywhere now. Will people start writing more like LLMs after reading so much of it?
I think once you focus on spotting AI writing, you start finding it constantly. The small-LLM work here is more interesting than the article's style.
People can accuse any clear technical prose of being AI-written now. It feels like you have to write sloppily to sound human.
I don't see the problem in this piece. I spend all day trying to decode actual LLM prose, and this does not read like that.
It reads like an LLM draft that was heavily edited. I use Claude for a first pass, rewrite it line by line in my voice, then ask Claude to fact-check the rewrites. If I cannot reword a claim myself, I probably do not understand it.
Pangram called it 50% human, which sounds plausible. I recently read Dostoevsky and noticed my own speech change for a while, so LLM prose influencing people makes sense.
I went from feeling bad about how coding changed to seeing LLMs as another way to build software. I built a Claude-assisted Excel/CSV parser for US property-tax compliance: deterministic code checks known layouts, then Haiku maps headers to support columns. Eventually I would train a tiny model for recurring cases and call Claude less.
I am on that trajectory too. LLMs keep experiments just beyond what I can absorb, and that tight feedback loop could work well for personalized education in some subjects.
I would like to see this repeated at around 1B parameters using the current small-model cookbook: gated delta nets, per-layer or n-gram embeddings, and gated residuals. I am not sure a roughly 2K context is long enough for gated delta nets to help.
I want an open-source trainer where I can choose an architecture such as Qwen or DeepSeek, a parameter count, and a dataset, then run it locally or on rented GPUs.
That goal sits between shared knowledge and convenience. There will be student projects and niche tools, but I do not want model training to become a market shaped mainly by what Microsoft thinks will sell.