๐คTogether AI BlogโขStalecollected in 34h
Serving DeepSeek-V4: Million-Token Inference Challenge
๐กMaster million-token serving on B200: KV compression & prefix caching insights
โก 30-Second TL;DR
What Changed
DeepSeek-V4 supports million-token context serving
Why It Matters
Advances efficient serving of long-context LLMs, enabling practitioners to handle massive inputs without performance drops. Highlights Together AI's leadership in inference infrastructure for cutting-edge models.
What To Do Next
Test DeepSeek-V4 endpoints on Together AI for million-token inference benchmarks.
Who should care:Developers & AI Engineers
Key Points
- โขDeepSeek-V4 supports million-token context serving
- โขOptimized inference on NVIDIA HGX B200 hardware
- โขCompressed KV layouts reduce memory usage
- โขPrefix caching and kernel maturity for efficiency
- โขEndpoint profiles tailored for long-context workloads
๐ง Deep Insight
AI-generated analysis for this event.
๐ Enhanced Key Takeaways
- โขDeepSeek-V4 utilizes a Mixture-of-Experts (MoE) architecture that dynamically activates a subset of parameters per token, which Together AI leverages to optimize compute-to-memory ratios during long-context inference.
- โขThe implementation on NVIDIA HGX B200 systems utilizes custom CUDA kernels specifically tuned for the Blackwell architecture's Transformer Engine, achieving higher throughput for FP8 precision operations compared to previous Hopper-based deployments.
- โขTogether AI's inference stack integrates speculative decoding strategies specifically adapted for million-token contexts, allowing the model to verify multiple tokens in parallel while maintaining accuracy for long-range dependencies.
๐ Competitor Analysisโธ Show
| Feature | Together AI (DeepSeek-V4) | Groq (Llama 3.3/Custom) | Fireworks AI (DeepSeek) |
|---|---|---|---|
| Context Window | 1M+ Tokens | 128K - 1M (varies) | 1M+ Tokens |
| Hardware | NVIDIA HGX B200 | Custom LPU | NVIDIA H100/B200 |
| Primary Focus | Long-context throughput | Ultra-low latency | High-speed production API |
| Pricing Model | Per-token / Reserved | Per-token / Reserved | Per-token / Reserved |
๐ ๏ธ Technical Deep Dive
- Architecture: DeepSeek-V4 employs a fine-grained MoE structure, significantly increasing the total parameter count while keeping active parameters per token low to manage latency.
- KV Cache Compression: Uses 4-bit/8-bit quantization for the Key-Value cache, allowing the storage of million-token sequences within the HBM3e memory limits of the B200 GPUs.
- Prefix Caching: Implements a multi-level caching strategy that stores KV states for common system prompts or document headers, reducing redundant computation across multiple requests.
- Kernel Optimization: Utilizes custom Triton-based kernels optimized for Blackwell's tensor core throughput, specifically targeting the memory-bound nature of long-context attention mechanisms.
๐ฎ Future ImplicationsAI analysis grounded in cited sources
Inference costs for million-token models will drop by 40% within 12 months.
Advancements in KV cache compression and hardware-specific kernel optimization are rapidly increasing the number of concurrent requests per GPU.
Long-context inference will shift from batch-processing to real-time streaming.
Improvements in prefix caching and speculative decoding are reducing the time-to-first-token (TTFT) for massive context windows, enabling interactive use cases.
โณ Timeline
2025-02
DeepSeek releases V3, establishing the MoE architecture foundation.
2025-11
DeepSeek-V4 is announced with native support for ultra-long context windows.
2026-03
Together AI announces support for NVIDIA Blackwell (B200) infrastructure.
2026-05
Together AI optimizes DeepSeek-V4 inference for million-token context on B200.
๐ฐ
Weekly AI Recap
Read this week's curated digest of top AI events โ
๐Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: Together AI Blog โ