๐ฆReddit r/LocalLLaMAโขStalecollected in 35m
DFlash Boosts Qwen 35B 33% on 8GB GPU
๐ก33% speedup for 35B MoE on 8GB RTX 2080โkey for local inference hackers
โก 30-Second TL;DR
What Changed
DFlash PR #22105 enables 35.6 tok/s on 24GB Q5_K_M model with 8GB VRAM
Why It Matters
Enables running massive MoE models on consumer GPUs, democratizing high-speed local inference for resource-limited setups.
What To Do Next
Build llama.cpp with DFlash PR #22105 and test on your low-VRAM GPU with -ncmoe 34.
Who should care:Developers & AI Engineers
Key Points
- โขDFlash PR #22105 enables 35.6 tok/s on 24GB Q5_K_M model with 8GB VRAM
- โขUses Qwen3.5-35B-A3B-DFlash-Q4_K_M draft (267MB) and -ncmoe 34
- โข33-34% speedup over baseline; draft-max 6 optimal at 99.3% accept rate
- โขCombines CUDA backend, MoE offload, no-mmap, -t 5
๐ฐ
Weekly AI Recap
Read this week's curated digest of top AI events โ
๐Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: Reddit r/LocalLLaMA โ