來源NVIDIA Developer Blog•較早收集於 20m
NVIDIA Dynamo Snapshot：加速 Kubernetes 推論工作負載啟動
#kubernetes#gpu-optimization#inference#scalingnvidia-dynamo-snapshotnvidiakubernetes
💡別再讓 GPU 空轉等待冷啟動。了解如何讓您的 Kubernetes 推論副本實現即時擴展。
⚡ 30 秒速覽
有什麼變化
縮短 Kubernetes 上推論工作負載的冷啟動時間
為什麼重要
此功能透過在自動擴展期間將延遲降至最低，直接提升了 LLM 和推論服務的營運效率。它使基礎設施團隊能夠在不超額配置昂貴 GPU 資源的情況下維持高效能。
下一步行動
評估您目前的 Kubernetes 推論擴展延遲，並測試 NVIDIA Dynamo Snapshot 以減少冷啟動開銷。
誰應關注：Developers & AI Engineers
關鍵要點
- •縮短 Kubernetes 上推論工作負載的冷啟動時間
- •防止彈性擴展期間的 GPU 空轉
- •提升流量高峰期間的服務層級協議 (SLA) 達成率
🧠 深度解析
背景與延伸：來自公開資料，非原文內容。引用 8 個來源。
🔑 增強重點摘要
- •NVIDIA Dynamo Snapshot leverages open-source CRIU (Checkpoint/Restore in Userspace) for host state and NVIDIA's proprietary
cuda-checkpointutility for GPU device state, enabling comprehensive serialization and restoration of AI inference workloads. - •The solution has demonstrated significant performance improvements, achieving up to a 21x reduction in startup time for large models like gpt-oss-120b, with restore times for single-GPU workloads approaching near-instantaneous speeds.
- •Key technical optimizations include deallocating KV cache memory before checkpointing to reduce size, utilizing parallel
memfdrestore and Linux native asynchronous I/O (AIO) for faster memory restoration, and a planned GPU Memory Service (GMS) to decouple model weights for concurrent restoration via high-bandwidth channels like GPUDirect Storage. - •Dynamo Snapshot is currently an experimental feature, supporting single-GPU vLLM and SGLang workloads, with NVIDIA actively working on integrating multi-GPU/multi-node support and compatibility with TensorRT-LLM in future releases.
- •Deployment on Kubernetes involves a privileged
snapshot-agentDaemonSet that runs on each node, orchestrating the checkpoint and restore processes forrunc-managed containers without requiring modifications toruncitself.
📊 競品分析▸ Show
| Feature/Approach | NVIDIA Dynamo Snapshot | AWS (EKS Best Practices) | BentoML | Mystic |
|---|---|---|---|---|
| Primary Mechanism | Checkpoint/Restore of full application (host & GPU) state. | Optimizing container image size, accelerating image pull (e.g., image streaming), optimizing inference server config, quantization. | Optimizing container image size, accelerating image pull, optimizing model weight loading. | Docker container optimization, container download speeds, model weights downloading, cloud/GPU provisioning optimizations. |
| Targeted State | Full runtime state (CPU memory, GPU memory, CUDA contexts, etc.). | Image download, model loading, server initialization. | Image download, model loading. | Image download, model loading, driver installation, server provisioning. |
| Startup Time Reduction | Up to 21x faster, restore times near 'speed of light' for single-GPU. | Aims to minimize delays from tens of seconds to minutes. | Achieved 25x faster cold starts in LLMs. | Aims to reduce 5-10 minute load times. |
| Kubernetes Integration | Kubernetes-native workflow via DynamoCheckpoint Custom Resources and snapshot-agent DaemonSet. | General Kubernetes best practices for Pod startup and scaling on EKS. | Focuses on deploying and scaling AI models with production-grade reliability on Kubernetes. | Addresses Kubernetes scaling (e.g., KEDA for scale-to-zero). |
| GPU Specifics | Directly checkpoints CUDA device state. | Focuses on efficient use of GPU resources, but not direct GPU state checkpointing. | Optimizes loading model weights into GPU memory. | Addresses GPU availability and provisioning. |
🛠️ 技術深入
- Checkpointing Mechanism: Dynamo Snapshot utilizes CRIU (Checkpoint/Restore in Userspace) to serialize the host-side state (CPU memory, threads, file descriptors, namespaces) and NVIDIA's
cuda-checkpointutility to dump the GPU device state (CUDA contexts, streams, device memory, virtual address mappings) to CPU memory. These two tools are composed to enable full inference worker state serialization. - Quiesce/Resume Hooks: Inference workers implement quiesce/resume hooks. The
quiescehook allows the workload to clean up non-checkpointable resources (e.g., deallocate KV cache memory) before checkpointing, optimizing checkpoint size. Theresumehook enables the workload to recreate resources that cannot be checkpointed in an established state, such as outbound TCP connections (due to changing pod IPs) and RDMA registrations, post-restore. - Kubernetes Integration: A privileged
snapshot-agentDaemonSet is installed via a Helm chart on each Kubernetes node. This agent handles the orchestration of CRIU dump andcuda-checkpointoperations forrunc-managed containers, integrating with the Dynamo Operator viaDynamoCheckpointCustom Resources. - Optimizations for Restore: Key optimizations to accelerate restore times include KV cache unmap (reducing checkpoint size), parallel
memfdrestore, and leveraging Linux native asynchronous I/O (AIO) to speed up memory restoration. - GPU Memory Service (GMS): Future plans include integrating a GPU Memory Service (GMS) to decouple large model weights from the process state. This allows for concurrent restoration of weights using high-bandwidth channels like GPUDirect Storage, further reducing restore latency.
- Prerequisites: Requires x86_64 (amd64) GPU nodes, NVIDIA driver 580.xx or newer (590.xx for multi-GPU snapshots), and ReadWriteMany storage for cross-node restore. It currently supports vLLM and SGLang backends.
- Limitations: Currently supports only LLM decode and prefill workers (not multimodal, embedding, or diffusion), is limited to single-GPU configurations (multi-GPU is experimental), cannot checkpoint active TCP connections, and is restricted to x86_64 architecture. The
snapshot-agentruns as a privileged DaemonSet.
🔮 前景展望基於引用來源的 AI 分析
Widespread adoption of Dynamo Snapshot will significantly reduce operational costs for AI inference deployments.
By enabling rapid scaling and minimizing GPU idle time during traffic fluctuations, organizations can avoid over-provisioning expensive GPU resources, leading to more efficient infrastructure utilization.
The planned multi-GPU and multi-node support will unlock more efficient deployment of extremely large language models (LLMs) and complex AI systems.
Extending Dynamo Snapshot's capabilities beyond single-GPU workloads will allow for broader applicability to sharded models and multi-component AI inference systems, which are currently limited.
Integration with TensorRT-LLM will further enhance performance and streamline the deployment of optimized LLM inference within the NVIDIA ecosystem.
Native support for TensorRT-LLM, a key NVIDIA library for LLM inference, would combine its performance optimizations with Dynamo Snapshot's fast startup, creating a more cohesive and efficient solution.
⏳ 時間線
2016
NVIDIA-Docker project initiated, enabling GPUs in the container runtime ecosystem.
2018-06
NVIDIA releases Kubernetes on NVIDIA GPUs Release Candidate, enabling GPU-accelerated deep learning on Kubernetes.
2024-04
NVIDIA acquires Run:ai, a Kubernetes workload management and orchestration platform for GPUs.
2025-05
NVIDIA Dynamo adds GPU Autoscaling, Kubernetes Automation, and Networking Optimizations.
2025-11
NVIDIA Grove, a Kubernetes API for orchestrating complex AI inference systems, becomes available within NVIDIA Dynamo.
2026-03
NVIDIA Dynamo 1.0, an open-source, production-grade inference operating system for AI factories, is released.
2026-05
NVIDIA Dynamo Snapshot is detailed, introducing checkpoint/restore for fast startup of inference workloads on Kubernetes.
📎 來源 (8)
Factual claims are grounded in the sources below. Forward-looking analysis is AI-generated interpretation.
📰
AI 週報
閱讀本週精選 AI 大事摘要 →
👉相關動態
AI 策展新聞聚合。所有內容版權歸原始發布者所有。
原始來源： NVIDIA Developer Blog ↗
每週電子報
每週一封，可隨時退訂。