Tag: #mobile-inference6 results
Google's Gemma 4 open models, including E2B/E4B, run locally on iPhones at over 40 tokens/sec with 128K context via MLX. Official Google AI Edge Gallery app enables easy deployment. Signals shift toward local AI eroding cloud token sales.
A new 1-bit 8B parameter LLM fits in 1.15GB memory and rivals Llama3 8B on benchmarks. It achieves 440 tok/s on RTX 4090, 136 tok/s on M4 Pro, and 40 tok/s on iPhone. The model is available on Hugging Face from a Caltech spinout.
A developer ported Microsoft's BitNet to iOS, achieving 45-46 tokens per second on iPhone 14 Pro Max with the 0.7B model using just 200MB memory. BitNet employs 1-bit weights (-1, 0, +1) for tiny, fast models. Plans include open-sourcing an instruction-tuned 2B model soon.
Post analyzes Google's TurboQuant KV cache compression (3-4 bits, zero loss) for local setups and mobile inference. Questions throughput gains, consumer GPU scaling, and phone RAM/battery impacts. Seeks benchmarks from mlx/llama.cpp forks.
The standard iPhone 18 may ship with 12GB of RAM, replacing an earlier rumor of 9GB. The additional memory could support more of Apple’s on-device AI features.
User tests Qwen3 9B at q4_0 quantization on Samsung S25 Ultra with 12GB RAM and Snapdragon 8 Elite. Achieves over 6 tokens/second generation speed using Hexagon NPU. Demonstrates viable on-device inference for mobile.