ggml ships v1.0.0: deprecated API pruned, FP8 and YaRN truncation added, speedups across CUDA, Metal, Vulkan and SYCL
- ggml 1.0.0 prunes the deprecated public API and removes the legacy DIAG_MASK_INF and DIAG_MASK_ZERO ops, and the RPC protocol moves to v9 so incompatible peers are rejected instead of silently misbehaving.
- New ggml_rope_set_truncate and ggml_rope_get_truncate control YaRN correction-bound rounding, and CUDA gains OCP FP8 support plus fused NVFP4 gate/up matmuls with GLU inside MMQ.
- CUDA adds few-row MMQ for batch sizes 2 to 24 and MMVF for BF16 few-row matmuls; Metal fuses MoE routing and keeps fusable node groups in one command buffer.
- Backend work spans a Vulkan prefill FlashAttention path, a device-side MoE scheduler on SYCL, decomposed Adreno FlashAttention in OpenCL, new WebGPU kernels and wider Hexagon op coverage.
- The scheduler now skips uploading inputs that are never read and prefetches only inputs produced by earlier splits, and the MoE cache can compute some cache misses on the CPU and copy results back asynchronously.
Hacker News 의견들
ggml and llama.cpp are the Linux of the AI world, driving local inference one bit at a time. Respect, and may you never see a day of low battery.
Long time llama.cpp user here, running on a crappy AMD card that only supports Vulkan, and it's still usable thanks to ggml. Great program, great project.
Imagine naming a library after your own initials. The GG is Georgi Gerganov.
What do you think Linux is named after?
Plenty of precedent for it. Debian, AWK, PKZIP, the Korn shell, dlmalloc. You build something to scratch an itch and the name is secondary.
What really impresses me is that they keep finding new ways to speed up the same hardware. I benchmarked before and after today's upgrade on a 50K token prompt and went from 4:22 to 3:59.
llama.cpp is a one-stop shop for getting almost any model onto almost any hardware, but a specialized engine with kernels built for your card will still be faster. If you just want something rock solid, it's one of the best.
Not just legacy hardware either, there's RISC-V support already. Congrats on 1.0.0.
Can we talk about the ggml.ai landing page? It looks like nobody tried.
The minimalism is the point. It loads as a single 1.33 KB request, fits on my phone screen, and tells you exactly what the project is.
I'll take that over another Claude-generated landing page with a beige gradient and enough JavaScript to make my laptop stutter while scrolling.
The MoE cache change is the one I care about. Load a model that slightly exceeds your VRAM and it keeps the cold experts in host memory instead of failing to load.