SourceReddit r/LocalLLaMA•Stalecollected in 5h
llama.cpp Hot Expert Cache Speeds MoE 27%
#moe#vram-cache#token-generationllama.cppllama.cppqwen3.5-122b-a10brtx-4090
💡27% faster MoE tokens on single 4090 via llama.cpp expert cache
⚡ 30-Second TL;DR
What Changed
Dynamic cache tracks hot experts every N tokens
Why It Matters
Unlocks faster single-GPU MoE inference for consumer hardware, bridging gap to unified memory systems. Critical for local deployment of massive models like Qwen3.5-122B.
What To Do Next
Clone github.com/ParmesanParty/llama.cpp and benchmark hot expert cache on your MoE model.
Who should care:Developers & AI Engineers
Key Points
- •Dynamic cache tracks hot experts every N tokens
- •22.67 tok/s on RTX 4090 (27% > layer offload)
- •44 VRAM slots, +45% vs all-CPU baseline
- •Repo: github.com/ParmesanParty/llama.cpp
📰
Weekly AI Recap
Read this week's curated digest of top AI events →
👉Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: Reddit r/LocalLLaMA ↗
This is a summary, not the original. Read the source, or get the weekly briefing.
The weekly digest
One email a week. Unsubscribe anytime.