Start and Restore GPU Workloads in Seconds with NVIDIA Snapshot | NVIDIA Technical Blog - NVIDIA Developer
NVIDIA has introduced Snapshot, an open-source component that reduces GPU workload initialization from minutes to seconds, dramatically improving AI inference scalability.
When inference demand spikes or a replica fails, available GPUs don't translate immediately into serving capacity. Initializing an LLM workload—a process known as a cold start—can take minutes, delaying scaling and recovery. NVIDIA Snapshot addresses this challenge by providing a checkpoint and restore system for GPU workloads on Kubernetes.
How Snapshot Works
Snapshot captures a fully initialized workload—including CPU and GPU memory—as a portable artifact and restores that state on compatible infrastructure, bringing inference capacity online in seconds instead of minutes. The process skips repeated workload initialization while infrastructure provisioning and container image loading remain outside the measured restore path.
The technology provides two operations for GPU Pods: Checkpoint, which captures initialized CPU and GPU memory as a portable artifact, and Restore, which starts a new Pod from that artifact on any node with the same GPU model and driver. Both operations use Kubernetes custom resources and require only a small handshake to pause before checkpointing and resume after restoring.
Performance Benchmarks
Design partners have demonstrated significant performance improvements with Snapshot. Crusoe reduced GLM-5.3 Pod readiness time from roughly 8 minutes to 31 seconds on NVIDIA HGX B300. Saturn Cloud benchmarked Snapshot for scale-from-zero inference, cutting vLLM startup from 2–8.5 minutes to 6–34 seconds on the same hardware.
Across all tested configurations, Snapshot brings a replica back in under 45 seconds, 8x to 36x faster than a cold start.
Technical Implementation
Benchmarks were conducted using NVIDIA B200 GPUs with driver version 615.71.09 and VAST (NFS) storage. Tests covered three inference engines: vLLM 0.31.0, SGLang 0.5.21, and TensorRT LLM 1.3.0. Models ranged from 600 million parameters to mixture-of-experts (MoE) models with 1 TB checkpoints across 8 GPUs.
The testing methodology compared cold start times—measured from process launch until engine initialization—with Snapshot restore times. Cold start measurements included startup, model loading, initialization, and compilation phases. Snapshot restore measurements covered agent setup, CRIU restore, CUDA restore, and wake/remap stages.
Key Findings
Benchmark results revealed four main findings: restore times stayed under 43 seconds across all configurations; restoring skips more than just weight loading; checkpointing pays off with the first restored replica; and for large MoE models, moving GPU state dominates the restore time.
Snapshot supports workloads spanning multiple GPUs on a single node as a single checkpoint artifact across ranks, enabling fast startup for large mixture-of-experts models using various parallelism strategies. The technology is available through the NVIDIA Snapshot repository, with installation instructions and usage guides for teams looking to implement it.
Related
