Sources: NVIDIA Technical Blog, "NVIDIA Dynamo Snapshot: Fast Startup for Inference Workloads on Kubernetes" (Saereesitthipitak, Feigin, Mailthody, Khadkevich et al., May 2026), and a maintainer's post after the November 2026 open-source release at KubeCon (Ron Kahn).
The problem
Scaling inference elastically on Kubernetes means cold-starting replicas. A cold start pulls the container, starts the engine, loads weights, warms kernels and captures CUDA graphs, then registers with the control plane. That can take minutes, during which GPUs are allocated but serve nothing, and traffic spikes breach SLAs.
How it works
Checkpoint both halves of the process. GPU-side state (CUDA contexts, streams, device memory, address mappings) is dumped to host memory with the CUDA driver's checkpoint capability (cuda-checkpoint). Host-side state (memory, threads, file descriptors, namespaces) is captured with CRIU.
Container-level checkpoints. Because CRIU state references the container's writable layer, the process tree and filesystem changes are captured together.
Node agent. A privileged DaemonSet (snapshot-agent, installed by Helm) checkpoints and restores runc containers on each node without modifying runc, and writes artifacts to shared storage.
Checkpoint at the right moment. The worker signals "ready for checkpoint" after the engine is warm but before it opens connections to the control plane, then waits in a polling loop. CRIU cannot capture live TCP connections, so restoring at that instruction lets the worker resume, see a "restore complete" signal, and only then join the cluster.
Shrink the image. The KV cache buffer is released before checkpoint and reallocated on restore. A proof-of-concept that stripes weights across local NVMe restores in under five seconds (21x faster start for gpt-oss-120b in NVIDIA's test).
Reported results
Per the maintainer: an upstream contributor took GLM-5.3 pod readiness from about 8 minutes to 31 seconds; another cut vLLM scale-from-zero startup from 2 to 8.5 minutes down to 6 to 34 seconds, both on HGX B300.
These are partner-reported numbers; the setup (storage, image size, model) drives most of the result.
Tradeoffs and failure modes
Privileged agent. A node-level DaemonSet with checkpoint rights is a high-value target; checkpoint artifacts contain model weights and any secrets in process memory.
Driver and hardware coupling. Restores generally need matching driver versions and GPU types; mixed fleets need per-variant snapshots.
Workload cooperation. Engines must expose a safe checkpoint point and re-establish external state after restore; naive checkpoints of live, networked workers fail.
Storage becomes the bottleneck. Restore speed is bounded by how fast artifacts reach the node.
Review checklist
How long do our inference replicas take from schedule to first token, and which phase dominates?
Can our engine reach a warm, not-yet-registered state where a checkpoint is safe?
Where would checkpoint artifacts live, who can read them, and are secrets excluded?
Are node driver versions uniform enough for restore?
Do autoscaling policies change if cold start drops from minutes to seconds?
Related entries
R086 (agent-adaptive serving, Evolving Infra), R085 (model routing), R027 (provider reliability), R060.