Jensen Huang Meets South Korean Tech Leaders Amid Market Dip

NVIDIA Blackwell Architecture (Successor to Hopper):

• Transistors & Process: 208 billion transistors, manufactured using custom TSMC 4NP process.
• Dual-Die Design: Features two reticle-limited dies connected by a 10 terabytes per second (TB/s) chip-to-chip interconnect, acting as a unified single GPU.
• Transformer Engine: Second-generation Transformer Engine with FP4 precision, optimizing performance and accuracy for next-generation models, doubling performance and memory support while maintaining accuracy.
• Tensor Cores: Fifth-generation Tensor Cores for AI compute, with native support for sub-8-bit data types (MXFP6 and MXFP4) to improve efficiency and accuracy in low-precision computations.
• Memory: Utilizes HBM3e memory, with up to 192GB and 8 TB/s bandwidth for B100/B200.
• NVLink: Fifth-generation NVLink, offering 1.8 TB/s per GPU.
• AI Management Processor (AMP): A dedicated RISC-V based scheduler chip on the GPU to offload scheduling from the CPU and better control GPU resources.
• Power & Cooling: Increased power demands (up to 1kW per GPU) necessitate a shift to liquid cooling for data center deployments.
• Rack-Scale Systems: GB200 NVL72 connects 36 GB200 Grace Blackwell Superchips (36 Grace CPUs and 72 Blackwell GPUs) in a liquid-cooled rack, delivering 30X faster real-time inference for trillion-parameter LLMs.

NVIDIA H100 GPU (Hopper Architecture):

• Transistors & Process: 80 billion transistors, TSMC 4N process.
• CUDA Cores: SXM variant has 16,896 CUDA Cores; PCIe variant has 14,592 CUDA Cores.
• Tensor Cores: Fourth-generation Tensor Cores (528 for SXM, 456 for PCIe) with FP8 Transformer Engine.
• VRAM: 80 GB HBM3 (SXM) or HBM2e (PCIe) with ~3.35 TB/s (SXM) or ~2 TB/s (PCIe) memory bandwidth.
• Throughput: Up to 3,958 TFLOPS (FP8 with sparsity) for SXM variant.
• NVLink: NVLink 4.0, 900 GB/s bidirectional bandwidth per GPU (SXM variant).
• Multi-Instance GPU (MIG): Up to 7 isolated GPU instances.
• Transformer Engine: Dynamically adjusts precision for LLMs to improve throughput while maintaining accuracy.

NVIDIA CUDA Platform:

• Definition: Proprietary parallel computing platform and API that allows software to use GPUs for accelerated general-purpose processing.
• Programming Model: Enables developers to program in languages like C++, Python, and Fortran, or leverage GPU-accelerated libraries and frameworks (e.g., PyTorch).
• Components: Includes the CUDA Toolkit (compiler, libraries, developer tools), CUDA-X libraries for higher performance across AI/HPC domains.
• Functionality: Facilitates parallel processing by allowing code (kernels) to execute across many GPU cores, managing memory hierarchy (global, shared, local memory) and data movement between CPU and GPU.