Nvidia Custom Chip for OpenAI Inference

🔑 Enhanced Key Takeaways

• •Nvidia's agreement with Groq is valued at $20 billion and includes an acqui-hire of founder Jonathan Ross, formerly of Google's TPU team, and other key engineers.[1][3]
• •The deal is a non-exclusive licensing of Groq's LPU inference technology, allowing Groq to continue operating independently under new CEO Simon Edwards.[2]
• •Groq's LPUs feature a 144-way VLIW tensor-streaming processor with only on-chip SRAM (230 MB per chip), optimized for low-latency single-user inference at batch size one.[1][3]

🛠️ Technical Deep Dive

• •Groq LPU uses a 144-way VLIW design built at GlobalFoundries, contrasting with Google's 8-way VLIW systolic arrays in TPUs, enabling cheap scaling but limited to 230 MB SRAM per chip with no external DDR or HBM.[3]
• •Static scheduling and tensor-streaming eliminate memory bottlenecks via ultra-fast on-chip SRAM, excelling in sequential low-latency tasks for real-time AI like chatbots.[1]
• •Running Llama 70B requires 10 racks and over 100 kW due to SRAM constraints; second-gen chip planned on Samsung 4nm for 2025 revenue but not yet evidenced in market.[3]

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline