ShardFlow Reaches 28 TPS Across WAN with Qwen2.5
💡A practical recipe for turning 86 ms WAN latency into 28 TPS with speculative decoding and CUDA Graphs.
⚡ 30-Second TL;DR
What Changed
On two T4 nodes in Iowa and Oregon, Qwen2.5-7B improved from 4.92 TPS without speculation to 20.31 TPS average and 28.10 TPS peak.
Why It Matters
ShardFlow demonstrates that model sharding over geographically separated, commodity GPUs can remain usable when speculative decoding amortizes network latency. The benchmark also highlights that GPU utilization and launch overhead—not only bandwidth—can dominate distributed inference performance.
What To Do Next
Clone ShardFlow and benchmark Qwen2.5-7B with K=8 speculative decoding and CUDA Graphs under your own WAN latency.
Key Points
- •On two T4 nodes in Iowa and Oregon, Qwen2.5-7B improved from 4.92 TPS without speculation to 20.31 TPS average and 28.10 TPS peak.
- •With K=8 drafting, speculative decoding commits about 4.07 tokens per WAN round trip instead of one token.
- •CUDA Graph capture reduced the 0.5B drafter forward pass from 112 ms to 25 ms by replacing roughly 1,500 Python-launched kernels with one replay call.
- •The system uses a zero-copy Rust TCP relay through Ohio, StaticCache, in-place KV rewind, and meta-device model slicing.
- •Qwen2.5-14B with NF4 4-bit quantization achieved 14.43 TPS average on the same two-node setup.
🧠 Deep Insight
Background and context from public sources — not the original article. 4 sources cited.
🔑 Enhanced Key Takeaways
- •Shardflow originated as a portfolio project specifically designed to enable distributed LLM inference on free, disparate cloud hardware like Kaggle notebooks.
- •The project evolved through three distinct architectural iterations, starting from a baseline of 2.27 TPS in v1 to the current 27.08+ TPS in v2.1.
- •The transition from DynamicCache to StaticCache was a critical engineering requirement to prevent memory pointer invalidation during CUDA Graph execution.
- •Shardflow is architecturally distinct from browser-based WebGPU P2P projects like 'Shard', focusing instead on high-performance GPU node orchestration.
- •The Rust-based TCP relay was implemented specifically to move the gateway out of the hot path, effectively decoupling the communication layer from the inference loop.
📊 Competitor Analysis▸ Show
| Feature | Shardflow | Petals | vLLM (Distributed) |
|---|---|---|---|
| Target Hardware | Free/Disparate Cloud | Decentralized/P2P | Data Center/Cluster |
| WAN Optimization | Rust TCP Relay/CUDA Graphs | BitTorrent-style DHT | NCCL/RDMA |
| Primary Use Case | Low-cost distributed inference | Large-scale model hosting | High-throughput serving |
| Latency Handling | Speculative Decoding | Pipeline Parallelism | Tensor Parallelism |
🛠️ Technical Deep Dive
- Architecture: Utilizes a master-worker pattern where the master node manages the speculative drafting process and the worker node handles the target model verification.
- Kernel Optimization: Employs CUDA Graph capture to consolidate ~1,500 individual Python-launched kernels into a single replayable graph, reducing CPU-to-GPU launch latency.
- Memory Management: Implements StaticCache to maintain fixed memory addresses for KV caches, which is a prerequisite for CUDA Graph stability.
- Networking: Uses a custom Rust-based TCP relay to minimize serialization overhead and maintain persistent connections across public WAN segments.
- Speculative Strategy: Employs a 0.5B parameter drafter model to generate K=8 token sequences, which are then validated in parallel across the WAN-connected nodes.
🔮 Future ImplicationsAI analysis grounded in cited sources
⏳ Timeline
📎 Sources (4)
Factual claims are grounded in the sources below. Forward-looking analysis is AI-generated interpretation.
Weekly AI Recap
Read this week's curated digest of top AI events →
👉Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: Reddit r/MachineLearning ↗
This is a summary, not the original. Read the source, or get the weekly briefing.
Weekly AI briefing
One email a week. Unsubscribe anytime.