Tag: #on-device-inference14 results
NVIDIA’s NeMo-Speech.cpp enables local, GGUF-quantized execution of speech recognition, text-to-speech, and codec components. The listed stack includes Magpie-TTS, Nemotron Speech, Nemotron ASR, Parakeet models, and NanoCodec.
The react-native-executorch library now supports running Gemma 4 models fully offline. It includes GPU acceleration via Vulkan on Android and MLX on Apple Silicon.
Wange Zhiyuan, a Tsinghua-led startup, raised tens of millions in funding to launch cPilot, an edge inference engine, and Amis, an AI platform. These tools aim to reduce hardware costs and Token consumption by optimizing local deployment of large models.
PyTorch introduces ExecuTorch as a cross-platform foundation for on-device audio inference in voice agents. It supports key workloads including transcription, real-time streaming, diarization, voice activity detection, and live translation. This unifies native inference for proliferating open-source voice models across devices.
LiveTranscriber is an open-source iOS app that runs Whisper, Qwen3-ASR, NVIDIA Nemotron Streaming, MOSS Multi-Speaker, and Qwen3 entirely on-device. It offers offline transcription, speaker separation, summaries, translation, Apple Watch recording sync, and searchable transcript history.
A developer demonstrated LFM2.5-2.6B running at approximately 17 tokens per second on a OnePlus 13 using only the phone’s CPU. The setup uses a Q4_K_M GGUF model, a custom 450 KB inference engine, and an ADB-based device probe suite, with a target of around 30 tokens per second.
MacPaw is partnering with Liquid AI to provide on-device inference capabilities for developers building apps for its app store. MacPaw is also developing a local version of its AI assistant, Eney, powered by Liquid AI models.
Developers are running LLMs locally on laptops, evading cloud-based security controls. Enabled by powerful hardware, quantization, and easy distribution, this creates blind spots for CISOs in integrity, provenance, and compliance. Traditional DLP fails to detect offline inference.
SpruceChat ports Qwen2.5-0.5B LLM to handheld gaming devices like Miyoo A30 using llama.cpp, running entirely on-device without internet. Model loads in ~60s on first boot with 1-2 tokens/sec generation speed. GitHub repo includes binaries for multiple devices.
Qwen released its new 0.8B model, which runs locally on a 7-year-old Samsung S10E at 12 tokens per second using llama.cpp and Termux. After resolving missing C libraries, it handles conversations and complex tasks effectively. This demo highlights powerful tiny LLMs on edge devices.