Deltafin streams 1.45 TB of Kimi K3 expert weights from four SSDs for 1 tok/s on an M5 Max MacBook Pro
- Deltafin reports sustained 1.00 tok/s decoding for the 2.78T-parameter Kimi K3 on a 128 GB M5 Max MacBook Pro, using its internal SSD and three Thunderbolt 5 SSD enclosures.
- Kimi K3's roughly 1.45 TB of expert weights cannot fit in memory, so Deltafin reads one 17.5 MB file for each selected expert, with 16 of 896 experts loaded per MoE layer.
- The project keeps expert weights at released MXFP4 precision but quantizes the resident attention trunk to int8; it claims token-identical output against its own reference prompt, not bit-exact output against BF16.
- A 512-token prompt takes about 6.3 minutes to reach first token because prefill currently has 6.2x read amplification; the author calls this the repository's largest open problem.
- Adding drives did not scale decoding linearly: the author reports one drive reached about 52% of the four-drive result, while RAID-0 lost because each MoE layer waits for its slowest expert read.
Hacker News opinions
The model is Kimi K3, 2.78T parameters with about 1.45 TB of expert weights, so it streams 16 of 896 experts per layer from four SSDs. On an M5 Max MacBook Pro with 128 GB, I measure 1.00 tok/s over 512 output tokens, but a 512-token prompt takes about 6.3 minutes before its first token because prefill rereads data 6.2 times too much.
I cannot read this as written. The wall of figures and jargon should be a gist or a much shorter explanation.
At 1 token/s, this very response would take roughly 17 minutes to generate. That makes 512 tokens about one question and one long answer.
Can this use an Intel or AMD machine with a large GPU and NVMe SSDs as virtual VRAM? I also want to know what speed the internal 4 TB SSD alone would get.
I do not see the use case. It is slow and expensive for something that performs this badly.
I agree it is useless for interactive chat. I use it for scheduled private work, such as reviewing numbers and drafting reports, where waiting is acceptable and the data cannot leave the machine.
The point is that a full frontier-like 2.8T model runs locally at all. Hacker projects do not need an immediate business case.
I could use a slow model as a classifier: feed it a large input and ask for one token, such as whether code has a bug. One token per second may be enough for that.
How are the SSDs physically connected?
They are in Thunderbolt 5 enclosures: one Gen4 and three Gen5 SSDs, alongside the MacBook's internal SSD.
Why not RAID-0 40 SSDs, or split the work across 40 MacBooks, to get 40 tok/s?
That does not scale linearly. Each MoE layer waits for the slowest of 16 expert reads; four drives raised decode from about 52% of the four-drive result to 100%, while RAID-0 made every read wait on every drive. Read scheduling and tail latency matter more.
Would Optane help because of its latency?
Probably not here. Each expert read is a 17.5 MB whole file, about 2.5 ms at 7 GB/s, so throughput rather than access latency dominates. More direct-attached drives and stable tail behavior help, but with diminishing returns.
I am more interested in using this approach for a smaller model such as GLM Flash. Kimi is an extreme demonstration.
The speed after filling a 200,000-token context would be the next concern.