Monokernel inference engine hits 3,300 tokens/s on MI300X
💡See how custom kernel engineering on AMD MI300X is pushing inference speeds to 3,300 tokens/s.
⚡ 30-Second TL;DR
What Changed
Achieves up to 3,300 output tokens/s per request on 8x MI300X.
Why It Matters
This demonstrates that AMD hardware can achieve extreme inference performance with custom kernel engineering, challenging the dominance of standard CUDA-based stacks.
What To Do Next
Visit playground.kog.ai to test the inference speed of their engine on the MI300X platform.
Key Points
- •Achieves up to 3,300 output tokens/s per request on 8x MI300X.
- •Runs full decode sequence as a single GPU-resident program.
- •Optimized by aligning memory access with physical die topology.
- •Currently supports 2B models with plans for frontier MoE support.
🧠 Deep Insight
Background and context from public sources — not the original article. 7 sources cited.
🔑 Enhanced Key Takeaways
- •Kog AI, a French company founded in 2023, develops a vertically integrated, real-time AI stack with a proprietary inference engine designed to bypass traditional abstraction layers.
- •The monokernel engine's optimization extends to grouping compute units by their associated Input/Output Die (IOD) to maximize the hardware's design performance.
- •Kog AI claims its engine achieves up to 3.5 times faster token generation compared to other leading inference solutions, including vLLM and TensorRT-LLM, across various top-tier GPUs (MI300X, H100, H200, B200), specifically emphasizing sequential decoding speed per individual request.
- •A critical technical innovation by Kog AI is the Kog Collective Communications Library (KCCL), engineered to achieve record-low cross-GPU communication latencies.
🛠️ Technical Deep Dive
- The monokernel inference engine executes the entire decode sequence as a single GPU-resident program on AMD ROCm datacenter GPUs.
- Optimization is achieved by precisely mapping memory access patterns to the physical die topology of the AMD MI300X.
- Compute units are grouped according to their associated Input/Output Die (IOD) to leverage the hardware's full design performance.
- The engine is built using optimized C++ and assembly code to eliminate inefficiencies and bypass traditional abstraction layers.
- Kog AI developed the Kog Collective Communications Library (KCCL) to specifically reduce cross-GPU communication latency.
- The reported 3,300 output tokens/s benchmark was achieved per request, with a batch size of 1, without speculative decoding or quantization, on an 8x MI300X setup, running a small 2B coding model.
- The AMD MI300X GPU itself features 192GB of HBM3 memory and 5.3 TB/s bandwidth, utilizing a chiplet-based architecture.
🔮 Future ImplicationsAI analysis grounded in cited sources
⏳ Timeline
📎 Sources (7)
Factual claims are grounded in the sources below. Forward-looking analysis is AI-generated interpretation.
Weekly AI Recap
Read this week's curated digest of top AI events →
👉Related Updates
AI-curated news aggregator. All content rights belong to original publishers.
Original source: Reddit r/MachineLearning ↗
This is a summary, not the original. Read the source, or get the weekly briefing.
The weekly digest
One email a week. Unsubscribe anytime.