Hugging Face Acquires GGML.AI
GGML.AI, key infrastructure for local LLM inference, has been acquired by Hugging Face. Announcement posted on r/LocalLLaMA. This move strengthens Hugging Face's position in efficient model deployment tools.
Search
Tag: #inference-engine11 results
⚙️
Major Tensor Fixes Improve CUDA Performance in ggml
The latest update to ggml (b9820) introduces significant performance improvements for CUDA by reducing synchronization requirements. These changes optimize tensor operations and async copies.
audio.cpp: Unified C++ Runtime for 12 Audio Models
A new native C++ inference framework built on ggml that consolidates 12 audio models into a single runtime. It significantly outperforms Python-based inference, achieving up to 5x speedups on CUDA.
⚙️
Safe GPU Inference in Rust with cuTile
cuTile Rust 引入了一種基於 Tile 的程式設計模型，利用 Rust 的所有權和借用檢查機制來確保 GPU 核心的記憶體安全與無資料競爭。研究團隊開發了 Grout 推論引擎，在 Qwen3 模型上展現了與 vLLM 和 SGLang 相當的效能。
Redis Founder Builds DeepSeek V4 Mac Engine
Redis founder has stepped in to create a dedicated inference engine for DeepSeek V4. This tool enables running the model locally on Mac computers. It brings efficient local LLM inference to Apple hardware.
⚙️
ZINC: Zig LLM Inference for AMD GPUs
ZINC is a new LLM inference engine written in Zig, enabling 35B models on $550 AMD GPUs via Vulkan. It loads GGUF models, achieves 7.1 tok/s on RDNA4, and addresses AMD consumer GPU gaps. Repo at github.com/zolotukhin/zinc.
⚙️
ik_llama.cpp 26x Faster Qwen 3.5 Prompts
ik_llama.cpp fork delivers 26x faster prompt evaluation (43 to 1,122 tok/s) and 3.5x generation speed on Qwen 3.5 27B Q4_K_M using RTX PRO 4000. Fused GDN kernels reduce graph splits from 34 to 2 for full GPU utilization. Pre-built Windows binaries available as drop-in replacement.
Krasis Hits 8.9x Prefill Speed vs Llama.cpp
Krasis LLM Runtime optimized for GPU-only prefill and decode, achieving 8.9x prefill and 4.7x decode over llama.cpp on single 5090. Supports large Qwen models like 122B-A10B with minimal RAM. Single-line GitHub install, OpenAI-compatible server planned.
Atlas Unlocks 102 tok/s on DGX Spark
Atlas, a pure Rust LLM inference engine with custom CUDA kernels for GB10's SM121 architecture, delivers 102 stable tok/s on Qwen3.5-35B-A3B, 2.3x faster than vLLM. It features 2-minute cold starts, tiny 2GB image, and excels on Qwen3-Next-80B-A3B at 82 tok/s. Solves DGX Spark's software issues for desktop petaflop compute.
⚙️
Hipfire: AMD-Optimized LLM Inference Engine
Hipfire is a new inference engine tailored for all AMD GPUs, featuring mq4 quantization. The creator shares quantized models on Hugging Face. Benchmarks on Localmaxxing reveal major speedups for RDNA3 and older cards.