Qwen3.8-Flash-Next Hits 181 Tokens per Second
๐กSee how two DGX Sparks sustain 181 aggregate tok/s for a 512K-context agent fleet.
โก 30-Second TL;DR
What Changed
Two DGX Spark nodes reached 181 tokens per second aggregate, with 30โ50 tokens per second for a single stream.
Why It Matters
The report suggests that relatively compact unified-memory AI systems can serve high-concurrency agent workloads when memory access and scheduling are carefully tuned. However, the headline throughput is aggregate rather than single-user latency and comes from a specialized, self-reported configuration.
What To Do Next
Benchmark your own vLLM deployment with prefix caching, MTP speculative decoding, and a pinned KV-cache budget before considering NVMe-mapped model tables.
Key Points
- โขTwo DGX Spark nodes reached 181 tokens per second aggregate, with 30โ50 tokens per second for a single stream.
- โขThe model runs at a stretched 512K context with a 2.89-million-token KV cache pool.
- โขNVMe-mapped n-gram embeddings, MADV_RANDOM, 64 gather threads, and prefix caching were key performance optimizations.
- โขThe deployment uses TP=2 over 200 Gb RoCE, MTP speculative decoding with three tokens, and vLLM.
๐ง Deep Insight
Background and context from public sources โ not the original article. 11 sources cited.
๐ Enhanced Key Takeaways
- โขQwen3.8-Flash-Next is a 125B-parameter multimodal Mixture-of-Experts (MoE) model that activates only 6B parameters per token.
- โขThe model introduces Gated DeltaNet (GDN) for history compression and Qwen Sparse Attention (QSA) for micro-block level context processing.
- โขIn SWE-bench Pro evaluations, the model achieved a score of 62.5, surpassing Claude Opus 4.6 Max (53.4).
- โขAlibaba reports that the training resource requirement for this architecture is approximately 1/9th that of the Qwen3.7-Plus model.
- โขThe model natively supports a 262K context window, which is extensible to 1M tokens using YaRN scaling.
๐ Competitor Analysisโธ Show
| Feature | Qwen3.8-Flash-Next | Claude Opus 4.6 Max | Qwen3.7-Plus |
|---|---|---|---|
| Architecture | 125B MoE (6B active) | Proprietary | Dense/MoE Hybrid |
| SWE-bench Pro | 62.5 | 53.4 | 55.8 |
| Context Window | 1M (Extensible) | 512K | 256K |
| Training Efficiency | 1/9th vs 3.7-Plus | N/A | Baseline |
๐ ๏ธ Technical Deep Dive
- Architecture: 125B parameter MoE with 6B active parameters per token.
- History Compression: Utilizes Gated DeltaNet (GDN) to manage long-term state.
- Attention Mechanism: Qwen Sparse Attention (QSA) processes context at micro-block granularity to minimize latency.
- Embedding Strategy: 51B-parameter N-gram embedding table capable of NVMe offloading for asynchronous access.
- Scaling: Employs YaRN (Yet another RoPE for Transformers) to extend native 262K context to 1M tokens.
๐ฎ Future ImplicationsAI analysis grounded in cited sources
โณ Timeline
๐ Sources (11)
Factual claims are grounded in the sources below. Forward-looking analysis is AI-generated interpretation.
Weekly AI Recap
Read this week's curated digest of top AI events โ
๐Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: Reddit r/LocalLLaMA โ
This is a summary, not the original. Read the source, or get the weekly briefing.
Weekly AI briefing
One email a week. Unsubscribe anytime.