๐คReddit r/MachineLearningโขStalecollected in 87m
Monokernel inference engine hits 3,300 tokens/s on MI300X
๐กSee how custom kernel engineering on AMD MI300X is pushing inference speeds to 3,300 tokens/s.
โก 30-Second TL;DR
What Changed
Achieves up to 3,300 output tokens/s per request on 8x MI300X.
Why It Matters
This demonstrates that AMD hardware can achieve extreme inference performance with custom kernel engineering, challenging the dominance of standard CUDA-based stacks.
What To Do Next
Visit playground.kog.ai to test the inference speed of their engine on the MI300X platform.
Who should care:Developers & AI Engineers
Key Points
- โขAchieves up to 3,300 output tokens/s per request on 8x MI300X.
- โขRuns full decode sequence as a single GPU-resident program.
- โขOptimized by aligning memory access with physical die topology.
- โขCurrently supports 2B models with plans for frontier MoE support.
๐ง Deep Insight
Web-grounded analysis with 7 cited sources.
๐ Enhanced Key Takeaways
- โขKog AI, a French company founded in 2023, develops a vertically integrated, real-time AI stack with a proprietary inference engine designed to bypass traditional abstraction layers.
- โขThe monokernel engine's optimization extends to grouping compute units by their associated Input/Output Die (IOD) to maximize the hardware's design performance.
- โขKog AI claims its engine achieves up to 3.5 times faster token generation compared to other leading inference solutions, including vLLM and TensorRT-LLM, across various top-tier GPUs (MI300X, H100, H200, B200), specifically emphasizing sequential decoding speed per individual request.
- โขA critical technical innovation by Kog AI is the Kog Collective Communications Library (KCCL), engineered to achieve record-low cross-GPU communication latencies.
๐ ๏ธ Technical Deep Dive
- The monokernel inference engine executes the entire decode sequence as a single GPU-resident program on AMD ROCm datacenter GPUs.
- Optimization is achieved by precisely mapping memory access patterns to the physical die topology of the AMD MI300X.
- Compute units are grouped according to their associated Input/Output Die (IOD) to leverage the hardware's full design performance.
- The engine is built using optimized C++ and assembly code to eliminate inefficiencies and bypass traditional abstraction layers.
- Kog AI developed the Kog Collective Communications Library (KCCL) to specifically reduce cross-GPU communication latency.
- The reported 3,300 output tokens/s benchmark was achieved per request, with a batch size of 1, without speculative decoding or quantization, on an 8x MI300X setup, running a small 2B coding model.
- The AMD MI300X GPU itself features 192GB of HBM3 memory and 5.3 TB/s bandwidth, utilizing a chiplet-based architecture.
๐ฎ Future ImplicationsAI analysis grounded in cited sources
Increased adoption of AMD MI300X for latency-sensitive AI applications.
Kog AI's demonstrated breakthrough in sequential decoding speed and low latency directly addresses critical needs for real-time, interactive AI and agentic workflows, making AMD's hardware more competitive in these demanding use cases.
Acceleration of advanced agentic AI workflows and complex chain-of-thought reasoning.
The engine's focus on fast sequential token outputs and overcoming latency bottlenecks directly enables more sophisticated, iterative agentic models that require rapid, deep reasoning capabilities.
Greater industry emphasis on hardware-software co-design for optimal AI inference performance.
Kog AI's success, stemming from deep optimization by aligning software with physical die topology and bypassing abstraction layers, highlights the significant performance gains achievable through tightly coupled hardware and software development.
โณ Timeline
2023
Kog AI founded in France.
2024-03-01
Kog AI raises Early Stage VC (Series 1) funding.
2025-07-14
Kog AI announces 3.5x breakthrough inference speed on AMD Instinct MI300X GPUs.
2025-11-06
Kog AI raises Accelerator/Incubator funding.
2026-05-28
Kog AI launches live preview of its Inference Engine on AMD ROCm datacenter GPUs, achieving 3,300 tokens/s on MI300X.
๐ Sources (7)
Factual claims are grounded in the sources below. Forward-looking analysis is AI-generated interpretation.
๐ฐ
Weekly AI Recap
Read this week's curated digest of top AI events โ
๐Related Updates
Same topic
Explore #inference
Same product
More on kog-ai-monokernel
Same source
Latest from Reddit r/MachineLearning
How AMD's 2006 ATI Acquisition Built Today's AI Empire
cnBeta (Full RSS)โขJul 25
๐ค
NeurIPS 2026 Theory Paper Review Distribution Discussion
Reddit r/MachineLearningโขJul 26
๐ค
Integrating AI Coding Agents with Cloud GPU Workflows
Reddit r/MachineLearningโขJul 26
Statistically-Lossless Quantization for Large Language Models
Reddit r/MachineLearningโขJul 24
AI-curated news aggregator. All content rights belong to original publishers.
Original source: Reddit r/MachineLearning โ