R096 · Infrastructure for agents at scale › Inference providers and serving reliability

NVIDIA Snapshot: checkpoint and restore GPU inference pods in seconds

Read the original

inference-servingkubernetescold-startcheckpointing

Sources: NVIDIA Technical Blog, "NVIDIA Dynamo Snapshot: Fast Startup for Inference Workloads on Kubernetes" (Saereesitthipitak, Feigin, Mailthody, Khadkevich et al., May 2026), and a maintainer's post after the November 2026 open-source release at KubeCon (Ron Kahn).

The problem

Scaling inference elastically on Kubernetes means cold-starting replicas. A cold start pulls the container, starts the engine, loads weights, warms kernels and captures CUDA graphs, then registers with the control plane. That can take minutes, during which GPUs are allocated but serve nothing, and traffic spikes breach SLAs.

How it works

Reported results

Tradeoffs and failure modes

Review checklist

  1. How long do our inference replicas take from schedule to first token, and which phase dominates?

  2. Can our engine reach a warm, not-yet-registered state where a checkpoint is safe?

  3. Where would checkpoint artifacts live, who can read them, and are secrets excluded?

  4. Are node driver versions uniform enough for restore?

  5. Do autoscaling policies change if cold start drops from minutes to seconds?

R086 (agent-adaptive serving, Evolving Infra), R085 (model routing), R027 (provider reliability), R060.

Connections

See R096 on the knowledge map

Part of the Research notes on nowucca.com. Figures and claims are the source's own unless the summary says otherwise.