NVIDIA details Kubernetes runtime recipe for Dynamo

NVIDIA said teams deploying Dynamo on Kubernetes need more than the serving framework itself: the GPU driver stack, scheduling, networking and observability layers must also be aligned before an inference endpoint is reliable. In a technical post published Oct. 30, 2026, the company described how its NVIDIA AI Cluster Runtime (AICR) packages Dynamo with validated infrastructure recipes to make that process repeatable.

Evidence and context

The post frames Dynamo as a distributed inference framework that coordinates model serving across GPU workers, but says the runtime beneath it is equally important. NVIDIA said a production inference cluster must confirm the GPU Operator and driver stack match the nodes, GPU nodes are labeled and discoverable, Kubernetes exposes GPUs through Dynamic Resource Allocation, the scheduler can place workers with the right topology and gang-scheduling semantics, node health signals are available before workloads land, Prometheus is reachable for metrics, and the Gateway API path is configured when Kubernetes-native ingress is used.

According to the article, AICR captures those requirements in a recipe. The recipe includes pinned component versions, compatibility constraints, node selectors, tolerations, checksums and validation expectations. NVIDIA said the Dynamo recipe ships version 1.4.2 of the dynamo-platform chart, including the Dynamo operator, runtime images and the nvidia.com/v1beta1 DynamoGraphDeployment custom resource definition. The company said v1beta1 replaces the former v1alpha1 API version.

The post also said the AICR recipe no longer installs NATS by default for Dynamo 1.4; instead, frontends, workers and optional Gateway API Inference Extension / Endpoint Picker Plugin components discover one another through Kubernetes resources. NVIDIA said NATS remains available as an opt-in for deployments that still need it.

Routing paths for Dynamo on Kubernetes

NVIDIA described two routing topologies. The first uses the Dynamo Frontend router behind a Kubernetes Service. In this path, the Frontend tokenizes requests, scores eligible workers using prefix overlap and current load, selects a worker and dispatches the request through Dynamo’s TCP request plane. The company said this is the smallest topology and is suitable for a model-specific endpoint or an existing edge proxy that only forwards traffic to the Dynamo Frontend Service.

The second path integrates Dynamo with the Gateway API and Gateway API Inference Extension. In that topology, a gateway such as agentgateway resolves an HTTPRoute to an InferencePool, the operator creates that pool from the DynamoGraphDeployment, and Dynamo EPP performs the token-aware endpoint selection. NVIDIA said the gateway remains the HTTP proxy, while a Dynamo Frontend sidecar in each routable worker pod runs in direct mode and forwards the request to the worker container.

The post said both paths use worker-published KV-cache events for routing decisions. It also noted that Dynamo can route based on predicted cache state, but prediction cannot see evictions, so routing quality can degrade under memory pressure. The AICR configuration in the article uses live events instead.

Deployment and validation workflow

NVIDIA laid out a five-step AICR workflow. First, a cluster snapshot is taken with the aicr snapshot command. Second, AICR generates a recipe for inference on the selected platform. Third, the runtime bundle is rendered into deployment-ready manifests and Helm values. Fourth, the runtime is deployed through Helmfile or other supported deployers. Fifth, the installed runtime is validated before the application workload is added.

For the Amazon EKS Dynamo inference recipe for NVIDIA GB200 NVL72 and NVIDIA GB300 NVL72 systems, the post listed pinned chart versions including NVIDIA GPU Operator v26.7.1, Node Feature Discovery 0.19.0, cert-manager v1.20.2, AWS EBS CSI Driver 2.59.0, AWS EFA v0.5.29, NVSentinel v1.25.0, NodeWright Operator v0.19.0, NVIDIA DRA GPU Driver 0.5.0, KAI Scheduler v0.16.9, Grove v0.1.0-alpha.13, Dynamo Platform 1.4.2, Agent Gateway v1.5.0, Agent Gateway CRDs v1.5.0, Kube Prometheus Stack 84.4.0, Prometheus Operator CRDs 28.0.1, Prometheus Adapter 5.3.0 and Kubernetes Ephemeral Storage Metrics 1.19.2.

Validation checks, according to the article, typically cover cluster baseline health, GPU Operator status, accelerator access, gang scheduling and autoscaling behavior, observability via Prometheus-backed metrics, and Gateway API and agentgateway readiness for Kubernetes-native ingress. NVIDIA said performance validation is separate because it deploys a temporary serving workload and consumes the GPUs under test.

The company closed by saying AICR and Dynamo are open-source components of the NVIDIA DSX AI Factory Platform, with deployment examples and community resources available through the NVIDIA DSX AI Factory GitHub and Slack community. It also pointed to hands-on workshop activity at KubeCon + CloudNativeCon North America, though the article did not specify a future catalyst date beyond that event mention.