Apple M3 Neural Engine DMA workaround raises Llama 3.2 1B decode from 10.0 to 24.3 tokens/s
- The author reports an Apple M3 Neural Engine throughput notch when a model's total weights are an integer multiple of 1 MiB, cutting DRAM weight streaming from 45-60 GB/s to 17-19 GB/s; 7 of ANEMLL's 15 models are affected.
- Avoiding the suspected KernelDMA speculative prefetch ring path raised Llama 3.2 1B decode on an M3 MacBook Air from 10.0 to 24.3 tokens/s, while measured DRAM use rose from 24.7 to 60.0 GB/s.
- The same split-4 change raised Qwen3-8B decoding from 1.36 to 2.97 tokens/s and DRAM weight throughput from 22.4 to 48.7 GB/s.
- A 2,747-observation sweep found sharp throughput dips at tensor dimensions that are multiples of 2048: D=2048 measured 16.93 GB/s, versus 44.5 GB/s at D=2016, with throughput recovering about 256 lines away.
- The transfers complete correctly, so the article describes a performance defect rather than a correctness failure and attributes the slowdown to a separate, credit-starved DMA issue regime.
Hacker News opinions
I want to know whether that SystemVerilog is a hypothetical model of the behavior or whether you have access to the actual RTL source.
The page hijacked my back button on a simple load. That is unacceptable behavior and should be fixed.
I am surprised browsers still permit this. They ought to block history manipulation like that at the browser level.
I open external links in a new tab and close the tab if a site pulls dark patterns. That avoids letting one site contaminate the history of the page I was reading.
I only saw the back-button problem in Safari. Edge and Firefox did not show it for me, though uBlock Origin may have affected the result.
The measurement is impressive, but I am not sure "erratum" is the right word. I usually see it as a correction notice for an error in a document.
M1 and M5 Max do not appear to be affected, according to ANEMLL's posted results. This may be specific to some Apple silicon generations.
The 2048-dimension resonance is striking. Ruling out core contention before testing address patterns makes the DMA explanation easier to follow.