DeepSeek-V4-Flash Runs Locally at 24 Tokens per Second
💡A practical data point for running a roughly 151GB MoE model at home with one RTX 5090.
⚡ 30-Second TL;DR
What Changed
The setup used an AMD Epyc 7663, 256GB DDR4-3200 ECC memory, and one RTX 5090 with 32GB VRAM.
Why It Matters
This result lowers the perceived barrier to running a large, capable MoE model locally without an expensive multi-GPU or data-center setup. Performance will vary significantly with quantization, memory bandwidth, context length, and workload characteristics.
What To Do Next
Reproduce the setup with your own prompt mix using UD-Q8_K_XL, then compare throughput and latency against DFlash and an RTX 3090 baseline.
Key Points
- •The setup used an AMD Epyc 7663, 256GB DDR4-3200 ECC memory, and one RTX 5090 with 32GB VRAM.
- •UD-Q8_K_XL delivered approximately 23.8–24.6 tokens per second across 100K–128K context workloads.
- •DFlash and a temporary RTX 3090 configuration were slower than the reported setup.
🧠 Deep Insight
Background and context from public sources — not the original article. 14 sources cited.
🔑 Enhanced Key Takeaways
- •DeepSeek-V4-Flash utilizes a 284B total parameter architecture with 13B active parameters per token, enabling high-efficiency inference for long-context tasks.
- •The model supports a native 1 million token context window, significantly exceeding the 128K context limit tested in the Reddit user's local setup.
- •The reported 24 t/s performance is currently constrained by the lack of full integration for speculative decoding frameworks like DSpark within local inference engines.
- •DeepSeek-V4-Flash serves as the efficiency-optimized counterpart to the 1.6T parameter DeepSeek-V4-Pro, specifically targeting agentic and reasoning workflows.
- •The model ecosystem expanded on August 21, 2026, with the release of DeepSeek-V4-Flash-Vision-Exp, introducing multimodal capabilities to the Flash architecture.
📊 Competitor Analysis▸ Show
| Feature | DeepSeek-V4-Flash | Llama-4-405B | GPT-5-Turbo |
|---|---|---|---|
| Architecture | 284B MoE (13B active) | Dense | Proprietary MoE |
| Context Window | 1M Tokens | 256K Tokens | 2M Tokens |
| Primary Use Case | Agentic/Reasoning | General Purpose | Enterprise/Multimodal |
| Local Inference | High (Multi-GPU) | Very High (Cluster) | N/A |
🛠️ Technical Deep Dive
- Architecture: Mixture-of-Experts (MoE) with 284B total parameters and 13B active parameters per token.
- Quantization: Performance relies on high-compression formats like UD-Q8_K_XL to fit within consumer-grade VRAM constraints.
- Inference Bottleneck: Current local execution is limited by CPU-based MoE offloading and lack of optimized speculative decoding support in llama.cpp.
- Hardware Requirements: Standard high-quality inference typically necessitates multi-GPU configurations (e.g., 4x RTX 3090s) due to the massive parameter count.
🔮 Future ImplicationsAI analysis grounded in cited sources
⏳ Timeline
📎 Sources (14)
Factual claims are grounded in the sources below. Forward-looking analysis is AI-generated interpretation.
Weekly AI Recap
Read this week's curated digest of top AI events →
👉Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: Reddit r/LocalLLaMA ↗
This is a summary, not the original. Read the source, or get the weekly briefing.
Weekly AI briefing
One email a week. Unsubscribe anytime.