Artificial Analysis benchmarks sub-8GB local AI models on iPhone 17 Pro and Galaxy S26 Ultra
- Artificial Analysis benchmarks small models directly on the iPhone 17 Pro and Galaxy S26 Ultra, each listed with 12GB of memory, using inference measurements collected with Liquid AI and independently validated by the site.
- The benchmark defines small models as models that fit within 8GB after quantization, including an 8K-context KV cache.
- Its mobile intelligence score is the simple average of five tests: BFCL subset, IFBench, AA-Omniscience, GPQA Diamond, and MATH-500, with results measured under a 16K-token context limit.
- End-to-end latency measures the wall-clock time to process a 1,024-token prompt and generate a 256-token response; the site says this excludes differences in model verbosity and turn count.
- The dashboard also records peak memory, 64K-context results where models fit and finish in time, and the share of evaluation generations that stop at the 16K-token limit.
Hacker News opinions
I like the benchmark, but it is heavily biased toward flagships. That is not very useful for a production mobile app targeting ordinary devices, where older base iPads may have only 4GB of RAM and Android midrange phones often lag iPhones on single-core performance and power efficiency.
Apple has been stingy with consumer RAM, and rising RAM prices could make phones with enough memory for meaningful ML workloads harder to build through 2030. I want private, distributed compute, but cloud LLM inference still looks much more viable for general use.
How do these models compare with GPT-4 or GPT-4o? I am curious whether phone-sized models have reached the frontier from a few years ago.
Qwen 3.5 9B scores 2 to 3 times higher than GPT-4o, depending on the version, on these benchmarks. That does not mean it is better for real use: I have seen benchmark winners fail badly while lower-scoring models work well for my tasks. Small local models are surprisingly good at coding and Bash agent tasks, but they still fall apart quickly in conversation, especially on knowledge and understanding.