High-speed NVMe arrays for large-scale LLM inference

🔑 Enhanced Key Takeaways

•NVMe arrays are increasingly utilized for Key-Value (KV) cache offloading in LLM inference, particularly for models with long context windows and multi-turn conversations, effectively extending memory capacity beyond GPU High Bandwidth Memory (HBM) and CPU DRAM.
•Specialized PCIe Gen5 switch adapters and fabrics are emerging to create direct, low-latency pathways between GPUs and NVMe storage, enabling GPU Direct Storage (GDS) and bypassing traditional CPU bottlenecks.
•Software frameworks like vLLM (with LMCache), DeepSpeed, and projects such as llm-d are actively developing and optimizing strategies for disk-based LLM inference, including tiered KV caching (GPU HBM, CPU DRAM, NVMe SSD) and disaggregated prefill/decode phases.
•While NVMe offloading significantly expands capacity, it introduces latency penalties compared to HBM. Performance gains are primarily derived from avoiding redundant computation through KV cache hits rather than NVMe raw speed matching HBM, necessitating optimizations like NVMe-direct paths and intelligent caching heuristics.
•Composable infrastructure platforms are gaining traction, allowing dynamic allocation and disaggregation of NVMe storage, GPUs, and other PCIe-connected resources over a fabric, which enhances resource utilization and flexibility for evolving AI workloads.

📊 Competitor Analysis▸ Show

Feature/Category	High-Lane-Count PCIe 5.0 NVMe Arrays (Self-Hosted)	Pure Storage FlashBlade (Enterprise Storage)	Liqid Composable Infrastructure	Groq (Specialized Hardware)
Primary Use Case	Overcoming VRAM limits for large LLM inference via local NVMe offloading.	High-performance, scale-out storage for AI/ML workloads, including RAG and KV cache acceleration.	Dynamic allocation and disaggregation of compute, memory, and storage resources over PCIe fabric.	Ultra-low latency, high-throughput inference using custom Language Processing Units (LPUs).
Hardware Focus	Direct-attached NVMe SSDs, PCIe 5.0 bifurcation, high-lane-count servers.	All-flash, scale-out storage arrays (FlashBlade//S) with NVMe-oF optimization.	Software-defined orchestration of commodity hardware (CPUs, GPUs, NVMe, CXL memory) via PCIe fabric.	Custom-built LPU chips optimized for sequential processing in LLMs.
Key Advantage	Cost-effective capacity extension for multi-terabyte models, high theoretical local throughput.	Optimized for parallel data delivery, metadata-indexed pipelines, and KV cache acceleration (Pure KVA).	Maximizes hardware utilization, provides real-time resource flexibility, and scales independently.	Exceptional speed and cost-efficiency for inference, often surpassing GPU-based systems.
Challenges/Limitations	Potential for I/O bottlenecks if not optimized (e.g., kernel page cache thrashing), complex software stack.	Enterprise-grade cost, may require integration with existing GPU infrastructure.	Requires specific hardware compatibility and software integration for full composability.	Limited ecosystem compared to GPUs, specific to inference workloads.
Performance (LLM Inference)	Theoretical 1.2TB/sec with 80 drives; actual performance depends on offloading strategy and software.	Up to 20x faster inference with Pure KVA for KV cache reuse.	Improved GPU utilization and reduced idle time through dynamic resource provisioning.	Delivers exceptional low-latency and high-throughput.

🛠️ Technical Deep Dive

PCIe 5.0 Bandwidth: A x16 PCIe 5.0 slot provides up to 64 GB/s of bidirectional bandwidth, doubling PCIe 4.0's capabilities.
PCIe Switch Architectures: Advanced PCIe switches, such as Marvell's Structera S 50256 (PCIe 5.0) with 256 lanes, offer 2048 GB/sec of aggregate bidirectional bandwidth. HighPoint's Rocket 7638D is a PCIe Gen5 switch adapter that provides dedicated x16 pathways for both GPUs and NVMe storage, enabling GPU Direct Storage (GDS) by bypassing the host CPU.
NVMe-Direct Paths: Frameworks like Dual-Blade implement NVMe-direct paths that bypass the operating system's filesystem and kernel page cache. This approach maps Key-Value (KV) tensors to contiguous logical block address (LBA) regions, reducing software overhead and improving latency for offloaded data.
Tiered KV Cache Management: A common strategy for LLM inference involves a three-tier storage hierarchy: hot KV blocks reside in GPU HBM, warm blocks in CPU DRAM, and cold or historical context blocks are offloaded to NVMe SSDs.
Offloading Mechanisms: DeepSpeed supports offloading optimizer states and model parameters from GPU VRAM to CPU RAM or NVMe. The dominant I/O pattern for NVMe offloading is typically large, sequential reads and writes, and RAID0 configurations can significantly improve throughput.
NVMe-over-Fabrics (NVMe-oF) with RDMA: Solutions like Dell's Project Lightning leverage NVMe-oF with Remote Direct Memory Access (RDMA) to facilitate direct data transfer from disk drives to GPU memory, minimizing CPU involvement and latency.
Compute Express Link (CXL): CXL, an open standard built on PCIe, provides memory and coherency extensions. CXL 1.0 coincided with PCIe 5.0, and newer versions like CXL 4.0 (based on PCIe 7.0, released in 2025) enable memory pooling and disaggregation, offering significantly lower latency than NVMe for memory expansion.

🔮 Future ImplicationsAI analysis grounded in cited sources

Composable infrastructure will become a standard for AI data centers.

The ability to dynamically allocate and disaggregate GPUs, NVMe storage, and CXL memory will be crucial for optimizing resource utilization and adapting to the rapidly evolving demands of LLM workloads and agentic AI.

NVMe-based KV cache offloading will be a mandatory feature for serving large context LLMs.

As LLM context windows expand to 128K tokens and beyond, GPU HBM alone cannot accommodate the KV cache for multiple concurrent users, making NVMe offloading essential for practical and cost-effective inference at scale.

Dedicated PCIe switch fabrics and GPU Direct Storage will be critical for high-performance AI.

Bypassing CPU bottlenecks and enabling direct, low-latency GPU-to-NVMe communication is necessary to fully leverage the increasing bandwidth of PCIe 5.0 and future generations for massive AI datasets and models.

⏳ Timeline

2019-05

PCIe 5.0 standard released, doubling bandwidth over PCIe 4.0.

2019

Compute Express Link (CXL) 1.0 specification released, building on PCIe 5.0 for memory and coherency.

2024-04

HighPoint Technologies announces industry's first 8-Channel PCIe Gen5 x16 NVMe Switch & RAID AICs, integrating Broadcom's PEX89048 switch IC.

2025-01

Dell introduces Project Lightning, leveraging NVMe-oF with RDMA for direct disk-to-GPU memory transfer for KV cache offloading.

2025-03

Marvell Technology introduces the Structera S family of PCIe switches, including the Structera S 50256 for PCIe 5.0 with 256 lanes.

2025-08

Pliops introduces the XDP LightningAI accelerator card for NVMe-based KV cache offloading, augmenting GPU server memory.

High-speed NVMe arrays for large-scale LLM inference

⚡ 30-Second TL;DR

Key Points

🧠 Deep Insight

🔑 Enhanced Key Takeaways

🛠️ Technical Deep Dive

🔮 Future ImplicationsAI analysis grounded in cited sources

⏳ Timeline

📎 Sources (30)

👉Related Updates

Innolight Clarifies 1.6T Demand Outlook

Apple M7 Ultra Rumors and Unitree G1 Robotics Milestone

ASUS ROG Launches 4K/FHD Dual-Mode OLED Monitor

The Reality of Using Valve's Steam Machine Today