NVIDIA Blackwell Tops STAC-AI LLM Inference Record

🔑 Enhanced Key Takeaways

• •TensorRT-LLM updates on Blackwell GPUs deliver up to 2.8x throughput improvement per GPU for DeepSeek-R1 MoE model inference over the past three months.[1][3]
• •Inference providers like Sully.ai and Latitude achieved 4x to 10x cost reductions on Blackwell by combining NVFP4 low-precision format, TensorRT-LLM, and open-source models versus Hopper.[2][4]
• •NVIDIA HGX B200 with eight Blackwell GPUs uses Multi-Token Prediction (MTP) and NVFP4 to boost DeepSeek-R1 inference performance in air-cooled setups.[1][3]
• •Fireworks AI on Blackwell enabled Sentient Labs to process 5.6 million queries in a week with low latency during high concurrency.[4]

🛠️ Technical Deep Dive

• •TensorRT-LLM optimizations include Programmatic Dependent Launch (PDL) to reduce kernel launch latencies and enhanced kernels utilizing Blackwell Tensor Cores.[1]
• •NVFP4 proprietary data format improves inference accuracy and throughput when activated across the full NVIDIA software stack including TensorRT-LLM.[1][2]
• •Multi-Token Prediction (MTP) increases throughput across various interactivity levels and sequence lengths on HGX B200 platform with eight Blackwell GPUs connected via fifth-generation NVLink.[3]
• •GB200 NVL72 platform features 72 interconnected Blackwell GPUs optimized for sparse MoE models like DeepSeek-R1.[1]

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline