SourceReddit r/LocalLLaMA•Stalecollected in 7h
Kimi K3 Hits 20 TPS on 16-GB10 Cluster
#multi-gpu-inference#throughput-benchmark#local-deploymentkimi-k3kimi k3dsparkvllmnvidia gb10
See how a full Kimi K3 deployment performs on 16 GB10 GPUs before planning your own serving cluster.
30-Second TL;DR
What Changed
Full Kimi K3 inference is running across a 16x GB10 cluster.
Why It Matters
This result suggests that large open models can be served at usable interactive speeds with a multi-GPU GB10 setup. Reproducible vLLM instructions could lower the barrier for teams evaluating Kimi K3 locally or in private infrastructure.
What To Do Next
Track the upcoming vLLM image and benchmark Kimi K3 on your own GB10 or multi-GPU setup using the same corpus and TPS metrics.
Who should care:Developers & AI Engineers
Key Points
- •Full Kimi K3 inference is running across a 16x GB10 cluster.
- •Reported throughput is 20 TPS average, 38 TPS peak, and 750 TPS for prefill.
- •The deployment uses DSpark, with further tensor-parallel optimization planned.
- •A vLLM image and setup instructions are expected after testing.
Deep Insight
AI-generated analysis for this event — not the original article.
Enhanced Key Takeaways
- •The GB10 cluster likely refers to NVIDIA's Blackwell-based B200 or GB200 NVL72 architecture, which is optimized for massive-scale inference of multi-trillion parameter models.
- •Kimi K3 is developed by Moonshot AI, a leading Chinese AI unicorn known for its long-context window capabilities, often competing with Google's Gemini 1.5 Pro.
- •DSpark, mentioned in the deployment, is an emerging distributed inference framework designed to handle tensor parallelism across high-bandwidth interconnects like NVLink Switch systems.
- •The 750 TPS prefill rate indicates significant optimization in KV cache management and attention mechanisms, likely utilizing FlashAttention-3 or equivalent kernel-level acceleration.
- •The community-driven effort to create a vLLM image suggests that Kimi K3 is being ported to open-source serving stacks to bypass proprietary API limitations.
Competitor Analysis
Inference Speed
- Kimi K3 (16x GB10)
- ~20 TPS (Avg)
- Llama 3.1 405B (H100 Cluster)
- ~15-25 TPS
- Gemini 1.5 Pro (TPU v5p)
- ~20-30 TPS
Context Window
- Kimi K3 (16x GB10)
- 2M+ Tokens
- Llama 3.1 405B (H100 Cluster)
- 128K Tokens
- Gemini 1.5 Pro (TPU v5p)
- 2M+ Tokens
Deployment
- Kimi K3 (16x GB10)
- Self-Hosted/DSpark
- Llama 3.1 405B (H100 Cluster)
- vLLM/TensorRT-LLM
- Gemini 1.5 Pro (TPU v5p)
- Proprietary API
| Feature | Kimi K3 (16x GB10) | Llama 3.1 405B (H100 Cluster) | Gemini 1.5 Pro (TPU v5p) |
|---|---|---|---|
| Inference Speed | ~20 TPS (Avg) | ~15-25 TPS | ~20-30 TPS |
| Context Window | 2M+ Tokens | 128K Tokens | 2M+ Tokens |
| Deployment | Self-Hosted/DSpark | vLLM/TensorRT-LLM | Proprietary API |
Technical Deep Dive
- Architecture: Kimi K3 utilizes a Mixture-of-Experts (MoE) or dense transformer architecture optimized for long-sequence retrieval and reasoning.
- Interconnect: The 16x GB10 configuration leverages NVLink Switch technology to minimize latency during tensor-parallel communication across nodes.
- Memory Management: The deployment utilizes PagedAttention (via vLLM integration) to manage the massive KV cache required for 2M+ token context windows.
- Optimization: The reported 750 TPS prefill speed suggests the use of speculative decoding or highly optimized kernel fusion for the attention layer.
Future ImplicationsAI analysis grounded in cited sources
Moonshot AI will release an official open-weights version of Kimi K3.
The community's focus on creating vLLM images and deployment instructions often precedes or forces an official release of model weights to maintain developer ecosystem relevance.
GB200-based clusters will become the standard for sub-second latency in long-context LLM inference.
The performance metrics achieved on the 16x GB10 cluster demonstrate that Blackwell architecture effectively removes the memory bandwidth bottlenecks previously limiting long-context model throughput.
Timeline
2023-10
Moonshot AI launches Kimi, its first long-context LLM.
2024-03
Kimi introduces support for 200,000 token context windows.
2024-05
Moonshot AI upgrades Kimi to support 2 million token context windows.
2025-11
Initial reports of Kimi K3 architecture development surface in technical forums.
2026-06
Moonshot AI begins private beta testing of Kimi K3 on Blackwell-class hardware.
- 2023-10Moonshot AI launches Kimi, its first long-context LLM.
- 2024-03Kimi introduces support for 200,000 token context windows.
- 2024-05Moonshot AI upgrades Kimi to support 2 million token context windows.
- 2025-11Initial reports of Kimi K3 architecture development surface in technical forums.
- 2026-06Moonshot AI begins private beta testing of Kimi K3 on Blackwell-class hardware.
Weekly AI Recap
Read this week's curated digest of top AI events →
Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: Reddit r/LocalLLaMA ↗
This is a summary, not the original. Read the source, or get the weekly briefing.
The weekly digest
One email a week. Unsubscribe anytime.