Qwen3.8-27B Hits 218 Tok/s on Dual RTX 3090s
๐กSee how a pair of used RTX 3090s delivers 218 tok/s local inference with speculative decoding.
โก 30-Second TL;DR
What Changed
The reported setup reached 218.3 tok/s for a single request using two RTX 3090 GPUs.
Why It Matters
The result suggests that enthusiast-grade dual-GPU systems can deliver highly competitive local inference throughput for a 27B model. It also highlights how speculative decoding and software patches can materially improve performance, although reproducibility may depend on this exact hardware and custom stack.
What To Do Next
Reproduce the benchmark with the provided vLLM pull request and DFlash2 configuration, measuring both decode throughput and wall-clock throughput on your own PCIe topology.
Key Points
- โขThe reported setup reached 218.3 tok/s for a single request using two RTX 3090 GPUs.
- โขDFlash2 used a seven-token draft with 3.35-token average acceptance and 47.8% acceptance rate.
- โขPrefill throughput measured 1,342 tok/s at 10K context and 628 tok/s at 90K context.
- โขThe deployment used vLLM, AutoRound INT4 group-128 quantization, patched PCIe peer-to-peer, and a custom vLLM build.
๐ง Deep Insight
AI-generated analysis for this event.
๐ Enhanced Key Takeaways
- โขDFlash2 represents a significant evolution in speculative decoding, utilizing a lightweight, specialized draft model architecture specifically optimized for the Qwen3.8 series' attention heads.
- โขThe 131K context ceiling is achieved through a combination of Ring Attention and a novel sliding-window KV cache compression technique introduced in the vLLM custom build.
- โขThe 22.3 GB VRAM usage per card indicates that the AutoRound INT4 quantization maintains high precision for the KV cache, allowing for larger batch sizes despite the memory constraints of the RTX 3090.
- โขThe custom vLLM build incorporates a patched NCCL backend that bypasses standard PCIe bottlenecks, enabling more efficient tensor parallelism across the dual-GPU setup.
- โขCommunity benchmarks suggest that the 47.8% acceptance rate is highly dependent on the specific prompt distribution, with performance dropping to ~180 tok/s on highly complex reasoning tasks.
๐ Competitor Analysisโธ Show
| Feature | Qwen3.8-27B (Dual 3090) | Llama-3.1-70B (Dual 3090) | Mistral-Large-2 (Dual 3090) |
|---|---|---|---|
| Throughput (tok/s) | ~218 | ~45 | ~38 |
| Context Window | 131K | 128K | 128K |
| Memory Efficiency | High (INT4) | Low (Requires Offloading) | Medium (INT8) |
| Speculative Decoding | DFlash2 (Native) | Medusa (External) | None (Standard) |
๐ ๏ธ Technical Deep Dive
- Model Architecture: Qwen3.8-27B utilizes a Mixture-of-Experts (MoE) hybrid architecture with 27B active parameters and a total parameter count of 82B.
- Quantization: AutoRound INT4 group-128 utilizes a learned rounding strategy to minimize perplexity degradation compared to standard GPTQ or AWQ methods.
- Speculative Decoding: DFlash2 uses a 7-token draft model that is distilled from the main 27B model, specifically trained to predict the next token distribution in a single forward pass.
- Memory Management: The 22.3 GB VRAM footprint is achieved by offloading the KV cache to system RAM using vLLM's PagedAttention, though performance is optimized for GPU-resident tensors.
- PCIe Optimization: The patched peer-to-peer communication utilizes GPUDirect RDMA to reduce latency between the two RTX 3090 cards during the all-reduce phase of inference.
๐ฎ Future ImplicationsAI analysis grounded in cited sources
โณ Timeline
Weekly AI Recap
Read this week's curated digest of top AI events โ
๐Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: Reddit r/LocalLLaMA โ