Skip to main content

NVIDIA GPU Stack

37 min read
AI documentation guide

The NVIDIA GPU software stack is organized in a layered structure for operating GPUs in Kubernetes environments.

Table 1. Role and core components by NVIDIA GPU software stack layer

LayerRoleCore Component
Infrastructure AutomationDeclaratively manage GPU drivers, runtimes, and pluginsGPU Operator (ClusterPolicy CRD)
MonitoringCollect GPU state and expose Prometheus metricsDCGM, DCGM Exporter
PartitioningShare a single GPU across multiple workloadsMIG, Time-Slicing
Inference OptimizationDatacenter-scale LLM servingDynamo, KAI Scheduler

This document covers the architecture and design decision criteria for each component. For GPU node provisioning (Karpenter), scaling (KEDA), and cost optimization, see GPU Resource Management.

GPU Operator Architecture​

Concept​

GPU Operator is an orchestration layer that bundles the entire GPU stack under a single ClusterPolicy CRD. Each component can be independently enabled/disabled, and GPU environments are automatically configured when nodes are added.

GPU Operator v26.3.2 (as of 2026-06)

Table 2. GPU Operator v26.3.2 bundled component versions

ComponentVersionRole
GPU Operatorv26.3.2GPU stack lifecycle management
NVIDIA Driver580.126.20GPU kernel driver
DCGMv4.5.2-1GPU monitoring engine
DCGM Exporterv4.5.3-4.8.2Prometheus metrics exposure
Device Pluginv0.19.2K8s GPU resource registration
GFDv0.19.2GPU node labeling
MIG Managerv0.14.2MIG partition auto-management
Container Toolkit (CDI)v1.19.1Container GPU runtime

v26.3.2 Key New Features: Kubernetes 1.36 support, DCGM Exporter Pod metadata fields (enablePodLabels/enablePodUID), DCGM Exporter custom annotations, NRI Plugin support for CRI-O 1.34+. Note: Blackwell GB200 NVL4 support was introduced in v26.3.0, CDI-by-default in v25.10.0, HPC Job Mapping in v25.10.x, and CDMM as a driver-level feature of the v25.3.2 era.

Component Structure​

Figure 1. GPU Operator ClusterPolicy component structure

Expand the diagram to read labels and follow connections.
Diagram

Component Roles:

  • Driver DaemonSet: Installs GPU kernel driver on nodes. Set enabled: false for AL2023/Bottlerocket as it's pre-installed in the AMI
  • Container Toolkit (CDI): Injects GPU devices into container runtime. CDI (Container Device Interface)-based for runtime independence
  • Device Plugin: Registers nvidia.com/gpu extended resource with kubelet. Enables kube-scheduler to place GPU Pods
  • GFD (GPU Feature Discovery): Exposes GPU model, driver version, MIG profiles as node labels. Used for nodeSelector/nodeAffinity
  • NFD (Node Feature Discovery): Exposes hardware features (CPU, PCIe, NUMA, etc.) as node labels
  • MIG Manager: Auto-applies MIG profiles based on ConfigMap. Reconfigures on node label changes
  • DCGM Exporter: Exposes DCGM metrics in Prometheus format

GPU Operator Configuration per EKS Environment​

Table 3. GPU Operator component installation by EKS environment

EnvironmentDriverToolkitDevice PluginMIGNotes
EKS Auto ModeNot installed (AWS auto)Not installed (AWS auto)Not installed (disabled via label)Not supportedDCGM/NFD/GFD work normally
Karpenter (Self-Managed)Not installed (AL2023 AMI)Not installed (AL2023 AMI)InstalledSupportedFull support
Managed Node GroupNot installed (AL2023 AMI)Not installed (AL2023 AMI)InstalledSupportedFull support
Hybrid Node (On-premises)Installed (required)Installed (required)InstalledSupportedGPU Operator required
AMI-specific GPU Driver Constraints
  • AL2023 / Bottlerocket: GPU driver pre-installed in AMI. Both driver and toolkit must be enabled: false
  • EKS Auto Mode: AWS auto-manages drivers. Device Plugin disabled via node label nvidia.com/gpu.deploy.device-plugin: "false"

GPU Operator on EKS Auto Mode​

On Auto Mode, AWS manages GPU drivers and Device Plugin, but GPU Operator installation is still useful when:

  • DCGM Exporter: GPU metrics collection (Auto Mode itself does not provide DCGM)
  • GFD/NFD: Per-GPU-model node labeling for nodeSelector usage
  • KAI Scheduler: Compatibility with projects depending on ClusterPolicy
# Auto Mode NodePool — Only Device Plugin disabled via label
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: gpu-auto-mode
spec:
  template:
    metadata:
      labels:
        nvidia.com/gpu.deploy.device-plugin: "false"
    spec:
      requirements:
        - key: eks.amazonaws.com/instance-family
          operator: In
          values: ["p5", "p4d"]
      nodeClassRef:
        group: eks.amazonaws.com
        kind: NodeClass
        name: default

Auto Mode Helm values keep the Device Plugin globally enabled and selectively disable it via node labels.

# GPU Operator Helm values for Auto Mode
driver:
  enabled: false          # Pre-installed in the AMI by AWS
toolkit:
  enabled: false          # Pre-installed in the AMI by AWS
devicePlugin:
  enabled: true           # Globally enabled, selectively disabled via node labels
dcgmExporter:
  enabled: true
  serviceMonitor:
    enabled: true
nfd:
  enabled: true
gfd:
  enabled: true

GPU Operator Configuration for Karpenter Nodes​

On Karpenter (self-managed) nodes, enable the full stack including MIG Manager. The GPU Operator ClusterPolicy does not accept a per-component nodeSelector, so to selectively disable an operand on specific nodes, use the nvidia.com/gpu.deploy.<component>: "false" node label. Auto Mode nodes already carry the Device Plugin disable label (nvidia.com/gpu.deploy.device-plugin: "false"), so the full stack is deployed only on Karpenter nodes. Operand tolerance of GPU node taints is set globally via daemonsets.tolerations.

# helm install gpu-operator nvidia/gpu-operator -f values.yaml
driver:
  enabled: false          # AL2023: Pre-installed in AMI
toolkit:
  enabled: false          # AL2023: Pre-installed in AMI
devicePlugin:
  enabled: true           # Globally enabled, selectively disabled via node labels
migManager:
  enabled: true
  config:
    name: default-mig-parted-config
    default: "all-balanced"
dcgmExporter:
  enabled: true
  serviceMonitor:
    enabled: true
    interval: 15s
nfd:
  enabled: true
gfd:
  enabled: true
# Operand DaemonSet tolerations are set globally, not per component
daemonsets:
  tolerations:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule

Key configuration points:

  • Operand placement is controlled by node labels (nvidia.com/gpu.deploy.<component>: "false"), not a per-component nodeSelector, because the ClusterPolicy template (both v25.10.1 and v26.3.2) does not render a nodeSelector value inside component blocks.
  • driver/toolkit: false — Pre-installed in the AL2023 AMI
  • migManager: true — Enables MIG functionality on Karpenter nodes. How MIG resources are exposed depends on the mig.strategy setting described under "MIG Pod request" below.

DCGM Monitoring​

Overview​

NVIDIA DCGM (Data Center GPU Manager) is a monitoring engine that collects GPU state and exposes metrics to Prometheus. GPU Operator automatically deploys DCGM Exporter as a DaemonSet.

Deployment Method Selection​

Table 4. DCGM Exporter DaemonSet deployment characteristics

ItemDetails
Resource Efficiency1 instance per node — minimal overhead
ManagementAuto-managed by GPU Operator
Metrics ScopeCollects all GPU metrics on the node
Suitable EnvironmentProduction environments (most cases)

Key GPU Metrics​

MetricDescriptionUsage
DCGM_FI_DEV_GPU_UTILGPU core utilization (%)HPA/KEDA trigger
DCGM_FI_DEV_MEM_COPY_UTILMemory bandwidth utilization (%)Memory bottleneck detection
DCGM_FI_DEV_FB_USED / FB_FREEFramebuffer used/free (MiB)OOM prevention, capacity planning
DCGM_FI_DEV_POWER_USAGEPower usage (W)Cost and thermal management
DCGM_FI_DEV_GPU_TEMPGPU temperature (C)Thermal throttling prevention
DCGM_FI_DEV_SM_CLOCKSM clock speed (MHz)Performance monitoring

Prometheus Integration Concept​

DCGM Exporter exposes Prometheus-format metrics at the :9400/metrics endpoint. Setting dcgmExporter.serviceMonitor.enabled=true during GPU Operator installation auto-creates the ServiceMonitor.

Collection chain:

GPU Hardware → DCGM Engine → DCGM Exporter (:9400) → Prometheus → Grafana/KEDA

Key design decisions:

  • Collection interval: 15s (default). For LLM serving, 10s recommended
  • Metrics filtering: Control cardinality by collecting only needed metrics via /etc/dcgm-exporter/dcp-metrics-included.csv
  • Pod-GPU mapping: Setting DCGM_EXPORTER_KUBERNETES=true adds pod, namespace, container labels to metrics

GPU Partitioning Strategies​

MIG (Multi-Instance GPU)​

MIG partitions Ampere/Hopper/Blackwell architecture GPUs (A100, H100, H200, B200) into up to 7 hardware-isolated GPU instances. Each MIG instance has independent memory, cache, and SM (Streaming Multiprocessor), guaranteeing stable performance without inter-workload interference.

MIG Core Value:

  • Hardware isolation: Memory, SM, L2 cache completely separated for QoS guarantee
  • Concurrent execution: Multiple inference workloads run simultaneously, each isolated from interference by the others; absolute throughput per instance is bounded by the SMs and memory the selected profile allocates, so a smaller profile delivers proportionally less than the full GPU
  • GPU Operator auto-management: MIG Manager auto-applies profiles based on ConfigMap

A100 40GB MIG Profiles:

ProfileMemorySM CountSuitable workload (BF16 weights)
1g.5gb5GB14≤2B weights, or short-context small models; 7B-class needs INT8/FP8 quantization
1g.10gb10GB14≤4B BF16, or 7B quantized to INT8/FP8
2g.10gb10GB28≤4B BF16, or 7B quantized; 2x the SMs of 1g.10gb for higher throughput
3g.20gb20GB42≤7B BF16, or 13B quantized
4g.20gb20GB56≤7B BF16, or 13B quantized; same memory as 3g.20gb, more SMs
7g.40gb40GB98≤13B BF16, or ~30B quantized; the full A100
Basis for model sizing

The "Suitable workload" column is a conservative weights-only lower bound, assuming BF16 weights occupy roughly 2 bytes per parameter (KV cache, activations, and fragmentation are additional). For example, a 70B model needs about 140 GB of BF16 weights alone, so it does not fit in a single 7g.40gb (40 GB) partition; serving 70B on a single A100 requires multi-GPU tensor parallelism or aggressive quantization. SM counts follow the A100-SXM4-40GB MIG profiles (1g = 14 … 7g = 98, i.e. 7 slices × 14 SMs). Actual throughput varies widely with the model, quantization (FP8/INT8/AWQ), batch size, sequence length, TP/PP configuration, and serving-engine version, so benchmark your target configuration before deployment. Relative comparison across profiles is governed by SM count and memory.

MIG Profile Management:

GPU Operator's MIG Manager watches node labels (nvidia.com/mig.config) and auto-applies MIG profiles. Define profiles in a ConfigMap, and MIG Manager reconfigures GPUs when node labels change. The ConfigMap name default-mig-parted-config below matches migManager.config.name in the Helm values above.

MIG reconfiguration is not disruption-free

Changing the MIG mode or geometry causes MIG Manager to terminate all GPU pods on the affected GPU before reconfiguring (NVIDIA GPU Operator MIG documentation). GPU workloads on that node stop during reconfiguration, and some environments (notably some CSPs) require a node reboot, so cordon/drain the node first. Reconfiguration only succeeds when no user workloads are running on the GPUs; if the prerequisites are not met, the nvidia.com/mig.config.state label is reported as failed. In production, perform this in a maintenance window and plan it together with a PodDisruptionBudget and a drain procedure.

# MIG Profile ConfigMap (mig-parted format)
apiVersion: v1
kind: ConfigMap
metadata:
  name: default-mig-parted-config
  namespace: gpu-operator
data:
  config.yaml: |
    version: v1
    mig-configs:
      all-1g.5gb:          # 7 small instances
        - devices: all
          mig-enabled: true
          mig-devices:
            "1g.5gb": 7
      mixed-balanced:      # Mixed configuration
        - devices: all
          mig-enabled: true
          mig-devices:
            "3g.20gb": 1
            "2g.10gb": 1
            "1g.5gb": 2
      single-7g:           # Single large
        - devices: all
          mig-enabled: true
          mig-devices:
            "7g.40gb": 1

How a Pod requests MIG devices depends on the Device Plugin's mig.strategy setting, specified as mig.strategy in the GPU Operator Helm values; the default is single.

  • mig.strategy: single (chart default): used when all GPUs on a node are partitioned into the same single profile. MIG devices are exposed as the generic nvidia.com/gpu resource, so a Pod requests:
# mig.strategy=single — MIG slices are exposed as nvidia.com/gpu
resources:
  requests:
    nvidia.com/gpu: 1
  limits:
    nvidia.com/gpu: 1
  • mig.strategy: mixed: used when different profiles coexist on one node. Only this strategy exposes per-profile nvidia.com/mig-<profile> resources, so for the request below to be valid you must set mig.strategy=mixed in the Helm values (the Karpenter values above do not set mig.strategy, so they default to single).
# mig.strategy=mixed — per-profile named resources are exposed
resources:
  requests:
    nvidia.com/mig-1g.5gb: 1
  limits:
    nvidia.com/mig-1g.5gb: 1

Time-Slicing​

Time-Slicing shares GPU computing time across multiple Pods based on time division. Unlike MIG, it's available on all NVIDIA GPUs but lacks inter-workload memory isolation.

Configuration:

GPU Operator's ClusterPolicy references a ConfigMap to enable Time-Slicing.

# Time-Slicing ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config
  namespace: gpu-operator
data:
  any: |-
    version: v1
    sharing:
      timeSlicing:
        resources:
          - name: nvidia.com/gpu
            replicas: 4    # Each GPU shared by 4 Pods

Pods request nvidia.com/gpu: 1 as usual. Time-Slicing oversubscribes the GPU by the configured replicas count — it only increases the advertised resource count and does not guarantee a fixed compute slice or memory quota per Pod. The sharing Pods run on the same GPU in time-sliced turns and contend for memory and compute.

MIG vs Time-Slicing Comparison​

ItemMIGTime-Slicing
Isolation levelHardware isolation (memory, SM, cache)Software time-sharing (no isolation)
Supported GPUsA100, H100, H200, B200All NVIDIA GPUs
Max partitions7 instancesAs set by replicas (not a hardware partition; contention grows)
Performance predictabilityGuaranteed (QoS)Varies with concurrent workload count
Memory safetyOOM does not affect other instancesOOM affects other workloads
Suitable environment RecommendedProduction inference, multi-tenantDev/test, batch inference
Time-Slicing Performance Characteristics
  • Scheduling model: GPU contexts are time-sliced and run in alternating turns per process. Unlike CUDA MPS, this is time division rather than concurrent kernel execution, so context-switching cost and queueing latency are added.
  • Performance variability: Throughput and latency depend on the number of concurrent workloads and each workload's kernel characteristics. Overhead and degradation vary widely with the configuration, so no fixed percentage can be guaranteed — measure with your target workloads.
  • No memory isolation: No memory quota is enforced, so one workload's OOM can affect others.
  • Suitable: Batch inference, dev/test environments | Not suitable: Real-time inference (SLA), high-performance training

Dynamo: Datacenter-Scale Inference Optimization​

Overview​

NVIDIA Dynamo is an open-source framework that optimizes datacenter-scale LLM inference. It supports vLLM, SGLang, and TensorRT-LLM as backends. In an independent benchmark (SemiAnalysis InferenceX), serving DeepSeek R1 on GB200 NVL72 with Dynamo (Disaggregated Serving + wide expert parallelism) reached up to 7x per-GPU throughput versus a B200 system without Dynamo — a comparison across different hardware, not a software-only speedup. NVIDIA's own same-hardware Disaggregated-vs-Aggregated comparison is in the 1.4x–2.5x range. Treat the 7x figure as workload- and hardware-specific, and benchmark your own configuration. Flash Indexer is a global indexer for KV cache-aware routing (170M ops/s), with a reported 2x TTFT improvement in the Baseten case. NVIDIA Run:ai (acquisition completed 2024-12-30) uses KAI Scheduler (Apache 2.0, open-sourced 2025-04) as its scheduler core.

Dynamo v1.2.x (as of 2026-06)
  • Serving modes: Aggregated + Disaggregated equally supported
  • Core technologies: Flash Indexer, NIXL, KAI Scheduler, Planner, EPP
  • Deployment: Kubernetes Operator + CRD (DGDR, DGD)
  • Latest version: v1.2.1 (2026-06-13), multimodal serving, KV routing improvements
  • License: Apache 2.0
  • Repository: ai-dynamo/dynamo

Core Architecture​

Dynamo supports both Aggregated Serving and Disaggregated Serving. In Disaggregated mode, Prefill (prompt processing) and Decode (token generation) are separated for independent scaling.

Figure 2. Dynamo Disaggregated Serving architecture

Expand the diagram to read labels and follow connections.
Diagram

Core Components​

Table 6. Dynamo core components and benefits

ComponentRoleBenefit
Disaggregated ServingSeparate Prefill/Decode workersIndependent per-phase scaling, maximize GPU utilization
Flash IndexerRadix tree-based per-worker KV cache indexingPrefix matching optimization, maximize KV reuse
KVBMGPU → CPU → SSD 3-tier cacheMaximize memory efficiency, support large-scale contexts
NIXLNVIDIA Inference Transfer LibraryUltra-fast GPU-to-GPU KV cache transfer (NVLink/RDMA). Shared by Dynamo, llm-d, production-stack, aibrix
PlannerSLO-based autoscalingProfiling → SLO target-based automatic Prefill/Decode scaling
EPPEndpoint Picker (Gateway API Inference Extension)An Inference Scheduler implementation that selects which endpoint to route a request to based on metrics; integrates with the K8s Gateway API Inference Extension
AIConfiguratorAuto TP/PP recommendationOptimal parallelization based on model size, GPU memory, network topology

llm-d Selection Guide​

llm-d and Dynamo both handle LLM inference routing/scheduling and overlap at the routing layer. They are not strictly mutually exclusive, however: both share the vLLM backend and NIXL KV transfer, and Dynamo can integrate llm-d as an internal component (see "Migration Path" below), making it closer to a superset. The table below compares the responsibilities and maturity each project emphasizes; choose based on operational complexity and the features you need.

llm-d:    Client → llm-d Router → vLLM Workers
Dynamo:   Client → Dynamo Router → Prefill Workers → (NIXL) → Decode Workers
Itemllm-dDynamo
ArchitectureAggregated + DisaggregatedAggregated + Disaggregated (equal support)
KV Cache RoutingPrefix-awarePrefix-aware + Flash Indexer (radix tree)
KV Cache TransferNIXLNIXL (NVLink/RDMA)
Pod SchedulingK8s default schedulerKAI Scheduler (GPU-aware)
AutoscalingHPA/KEDA integrationPlanner (SLO-based) + KEDA/HPA
BackendvLLMvLLM, SGLang, TRT-LLM
ComplexityLow — add router to existing vLLMHigh — replace entire serving stack
Maturityv0.5+v1.0 GA

Table 7. Recommended llm-d/Dynamo choice by scenario

ScenarioRecommendation (heuristic)
Add routing to existing vLLMllm-d
Small cluster, prioritize minimal operational complexityllm-d
Gateway API-based K8s nativellm-d
Large cluster, throughput maximization is the priorityDynamo
Long context (128K+) workloadsDynamo (3-tier KV cache)
Fast adoption, low operational complexityllm-d

The "scale" distinctions above are heuristics, not fixed GPU-count thresholds. The actual crossover point depends on model size, traffic pattern, the gain from disaggregation, and operational capacity, so validate it in your target environment.

Migration Path

Starting with llm-d and transitioning to Dynamo as scale grows is practical. Both share vLLM backend and NIXL KV transfer. The key differences are Dynamo's Flash Indexer, KAI Scheduler, and Planner. Dynamo 1.0 can integrate llm-d as an internal component, making it viewable as a superset rather than a complete alternative.

KAI Scheduler​

KAI Scheduler is NVIDIA's GPU-aware Kubernetes Pod scheduler. Unlike the default kube-scheduler, it recognizes GPU topology (NVLink, PCIe), MIG slices, and Gang Scheduling to determine optimal Pod placement.

Core Features​

Table 8. KAI Scheduler core features

FeatureDescription
GPU Topology AwarenessMinimizes communication cost by recognizing NVLink/PCIe connection structure
MIG-aware SchedulingRecognizes MIG slices as individual scheduling units
Gang SchedulingGuarantees all Pods are placed simultaneously in distributed training
Fair-share SchedulingPer-namespace/team GPU quota management
PreemptionPriority-based Pod replacement

Design Considerations​

  • ClusterPolicy dependency: KAI Scheduler requires GPU Operator's ClusterPolicy to be installed
  • EKS Auto Mode: KAI Scheduler usable after installing GPU Operator with Device Plugin disabled via label
  • Relationship with kube-scheduler: KAI Scheduler does not replace kube-scheduler; it operates as a Secondary Scheduler delegated only for GPU workloads
KAI Scheduler ≠ Autoscaling

KAI Scheduler is a scheduler that decides which node to place Pods on. It is separate from autoscaling (KEDA/HPA) that increases Pod count or provisioning (Karpenter) that adds nodes.

References​