PrismML ships Ternary Bonsai 2 27B: 5.9GB footprint, 98.2% of Qwen3.8 27B performance

Love this for the 16GB graphics card crowd. Qwen3.8 27B has been incredible but it needs 32GB to really run, so I'm trying this on my 16GB Intel B50. Not sure the XPU cores can accelerate these quants yet, but maybe in time.

You can already run the ~4-bit quants on 24GB if you're not too picky about context size. This would be a very surprising jump in performance at the size they claim, so I want to see more benchmark detail.

Tried it today on a B70 and couldn't get anything usable out of it. Prism's llama.cpp fork only has kernels for CUDA, CPU and Vulkan. No SYCL at all.

I don't follow the local model scene that closely, but this seems amazing. Can it run on Apple silicon too?

It says up to 143 tokens/second on an RTX 5090 and 46.8 on an M5 Max, so yes. And on a 4090 it's 0.714 mWh/token, 40% more efficient than an 8B in full precision.

Their first 27B Bonsai ran on an iPhone, so Apple silicon is not the question here.

These are small enough to run entirely in the browser, there's a WebML space on HuggingFace for it. Just clear the downloaded weights afterward. Like the last model it's amazing it works as well as it does, but use it for anything long and it falls apart spectacularly.

I'd love to see a Bonsai start from a 100B+ parameter model and get that under 30GB. Though at that point we'd have to call it Topiary.

Hoping they release an 8B v2 off the Qwen 3.8 series, that would be a genuinely powerful model running straight on phones.

That requires Alibaba releasing a Qwen 3.8 8B first, so don't hold your breath.

If you try the GGUFs, be aware you need Prism's llama.cpp fork to get them to load. I got about 20 tokens/second on an M5 Pro, then after a server restart I got 44 and I'm not sure why.

I used that setup to generate an SVG of a pelican riding a bicycle and it took 18 minutes 20 seconds. Pretty decent for a 5.5GB model file.

Where did those instructions come from? They have a demo repo with a setup.sh script, and the release tag and weight file you suggested don't match what they wrote.

It would be great to get upstream llama.cpp support for this instead of everyone running a vendor fork.

Thanks for all the public exploration. The fix I proposed was merged roughly 49 commits after the PrismML fork. The 'tensor API is not supported' warning happened because llama.cpp's startup probe couldn't compile a matmul2d kernel, since Metal's tensor headers need language version 4.0 and ggml-metal-device.m omitted MTLCompileOptions.languageVersion.

I really wish people would stop saying N times smaller. That makes no sense, it's 1/9th the size. You don't get a smaller quantity by multiplying by a number greater than 1.

For speed we say 9x faster and nobody blinks at it, so I think 9x smaller is perfectly reasonable.

I just mentally switch to the reciprocal when I read it, it's easier than convincing the world to stop. Same thing happens with 9x faster, where you're actually measuring time.

They probably rephrased it from something like 'we compressed the model by a factor of 9', where the number measures the transformation the model went through, not the size of the result.