Intelligence per Watt: local LMs answer 88.7% of 1M queries as efficiency rises 5.3x since 2023

Intelligence per watt is a stupid metric. It's not because a model performs better that it necessarily needs more energy or compute.

I don't think that's what they're saying. They estimate the value across different architectures to find the more efficient ones, and they use open models to remove unknowns. They're not advocating one model over another.

Unless I misread it, are they saying local GPUs use less energy? That's surprising and almost unbelievable because of batching. Local is usually not batched at all.

Small models are much smaller than frontier models, which is how they end up consuming less energy despite the low batch count. And with local models getting strong agentic capabilities, batching becomes a reality locally too.

You misread it. The 1.4x is on intelligence per watt, not raw energy, so local is roughly 40% dumber per watt than cloud on identical models. Also they seem to ignore the idle baseline costs and focus on the accelerator under load. CPUs, memory, PSU losses, network, fans and cooling add a cost floor that favors data centers.

This is the metric I've been waiting for. We run everything local (ollama plus neo4j) for compliance reasons, so quality per watt is literally our budget line. qwen2.5:3b on an M2 MacBook handles nl-to-cypher for simple graph schemas in 3 to 5 seconds and the energy cost is a rounding error next to shipping the same queries to a frontier API. The hard part was never the model, it was parsing PDFs locally without a vision model.

Incredibly important research. Local LLMs are good enough now. It takes less time for a local model to take the first action on your task than for Claude to validate your login and put you in queue. Local models are persistent and predictable, and distributing demand is better for the power system. In winter the GPU doubles as a 300W in-house heater.

That's only true if your local model is already resident in RAM or VRAM.

I saw measurements on SBC NPUs (3588) with a decent power win over CPU, but also a perplexity loss relative to CPU. This will be hard to quantify reliably in practice.

We publish live energy stats for all models on our service at portal.neuralwatt.com/energy-pricing. Model routing per request instead of per session breaks down on energy and cost, since keeping the KV cache warm cuts both the time component and the expensive prefill energy. A cache-aware local/cloud hybrid serving setup is where I'd put the effort.