DFlash2 Hits 138 TPS on RTX 3090
π‘See how software optimizations push Qwen3.8-27B to 138 TPS on a 250W RTX 3090.
β‘ 30-Second TL;DR
What Changed
Single-user speed improved to about 138 TPS on an RTX 3090 limited to 250 watts.
Why It Matters
This update suggests that careful quantization, speculative decoding, and cache management can deliver major local-inference gains without newer hardware. It is especially relevant for teams serving long-context workloads on consumer GPUs.
What To Do Next
Benchmark Qwen3.8-27B with DFlash2 and vLLM 0.27.1 on your RTX 3090, comparing latency with and without mamba prefix caching.
Key Points
- β’Single-user speed improved to about 138 TPS on an RTX 3090 limited to 250 watts.
- β’DFlash2 increased speculative decoding efficiency from 2.8 to 3.3 tokens per step.
- β’Lookup-augmented drafting improved repeated-content workloads by up to 29% tokens per step.
- β’Hybrid-model prefix caching reduced a 24k-token document follow-up from 23 seconds to 0.85β1.35 seconds.
- β’The stack supports 64k context with additional allocator fixes for the hybrid model.
Weekly AI Recap
Read this week's curated digest of top AI events β
πRelated Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: Reddit r/LocalLLaMA β
Weekly AI briefing
One email a week. Unsubscribe anytime.