llm-d Based EKS Distributed Inference Guide
Reviewed baseline: llm-d v0.8.1 (released 2026-06-26; documentation reviewed 2026-09-18). Components: llm-d Router v0.9.0, GIE v1.5.0, Gateway API v1.5.1. Joining CNCF Sandbox in March 2026 is a separate milestone from this release.
Overview
llm-d is an Apache 2.0-licensed Kubernetes-native distributed inference stack and a CNCF sandbox project founded by Red Hat, Google Cloud, IBM Research, CoreWeave and NVIDIA. It combines the vLLM inference engine, an Inference Gateway connecting a compatible proxy to an EPP, and Kubernetes Gateway API to provide intelligent inference routing for large language models.
A load balancer that ignores cache state, such as round-robin, spreads requests that share a prefix across vLLM Pods, so each Pod can reuse only the prefixes it has computed itself. llm-d's endpoint picker (EPP) instead scores each candidate Pod: the optimized-baseline configuration combines a prefix-cache scorer, which estimates how much of the request's prefix each Pod can reuse, with load-aware scorers for KV cache utilization and queue size. When traffic shares long reusable prefixes, this can shorten Time To First Token (TTFT) and save prefill computation; the gain depends on how much of the traffic shares reusable prefixes, on load and on the configured scorer weights. For llm-d's published comparison with round-robin and its conditions, see KV Cache Optimization.
For llm-d EKS deployment design and cluster preparation, see the Custom Model Deployment Guide.
llm-d v0.8.1 Gateway Mode integrates an existing Gateway API implementation with an EPP (Endpoint Picker). One compatible Gateway can route to both traditional Services and InferencePools. The EPP selects endpoints; it is not a second traffic proxy.
- Single Gateway: The selected implementation and version must support
HTTPRoute → InferencePooland EPP external processing integration. Check its policies and configuration for TLS, authentication, and rate limiting. - Separate edge and inference Gateways: Choose this when retaining an existing ingress or separating security boundaries, ownership, or scaling. Account for the extra proxy hop, latency, cost, and consistent timeout, retry, streaming, authentication-header, and observability handling.
Two Gateways are not a universal llm-d deployment requirement. The diagram below shows a single Gateway configuration.
llm-d's 3 Well-Lit Paths
The table summarizes three representative patterns. The complete v0.8.1 Well-Lit Paths are organized into Foundations and Workloads; Optimized Baseline is the current starting point for Intelligent Inference Scheduling.
Architecture
llm-d's Intelligent Inference Scheduling architecture is composed as follows.
Solid lines show requests/responses and ext-proc calls; dotted lines show configuration references and Pod management. GPU counts and TP are illustrative workload values. InferencePool selects existing Pods and references an EPP; images, GPU requests, and replica counts belong to workloads such as Deployment/LeaderWorkerSet.
Figure 1. Component composition and reference relationships of llm-d Intelligent Inference Scheduling
llm-d vs Traditional vLLM Deployment Comparison
| Feature | Traditional vLLM Deployment | llm-d Deployment ✨ |
|---|---|---|
| Routing Method | Round-Robin / Random | KV Cache-aware Intelligent Routing |
| Gateway Integration | Separate Ingress/Service configuration | Native Gateway API integration |
| Scaling Management | Manual HPA configuration | Workload/autoscaler manages replicas; InferencePool discovers Pods |
| KV Cache Utilization | Independent management per Pod | Routes to cached Pods; transfer/offloading requires separate configuration |
| Installation Method | Combining individual Helm charts | v0.8.1 recipe: router Helm chart + model-server Kustomize |
| Model Definition | Writing Deployment YAML directly | Deployment/LeaderWorkerSet defines image, GPUs, and replicas; InferencePool selects endpoints and InferenceObjective sets request priority |
Gateway API CRD
This guide uses the following Gateway Mode resources. Gateway API/GIE CRDs and the optional llm-d InferenceObjective CRD have different owners and installation packages. Baseline schemas: llm-d v0.8.1, InferencePool v1, InferenceObjective v1alpha2, HTTPRoute v1.
Default Deployment Configuration
This configuration follows the NVIDIA GPU example in Optimized Baseline v0.8.1. These are recipe values, not EKS-validated defaults or universal defaults for every deployment path. This release uses a router Helm chart and model-server Kustomize manifests.
Table 1. Default deployment configuration based on the Optimized Baseline v0.8.1 example
| Setting | Example value | Defined in |
|---|---|---|
| Model | Qwen/Qwen3-32B | Model-server arguments |
| vLLM image | vllm/vllm-openai:v0.23.0 | Workload container image, per the release component table |
| Replicas | 8 | Workload replica configuration |
| Tensor Parallelism / GPU | TP=2 / 2 GPUs per replica | Model-server arguments and Pod GPU requests |
| Total GPUs | 16 | 8 replicas × 2 GPUs; node placement is a separate decision |
InferencePool selects this workload's Pod labels. InferenceObjective expresses request policy and does not replace these deployment settings.
Qwen3-32B Model Selection Rationale
Qwen3-32B is the default model in the Optimized Baseline example above and is Apache 2.0-licensed for free commercial use. Its 32.8B BF16 weights occupy roughly 66GB (32.8B × 2 bytes), and KV cache and runtime buffers require additional VRAM on top of that. TP=2 across two H100 80GB GPUs gives about 160GB aggregate, leaving headroom for KV cache; whether that configuration serves a given workload stably depends on context length, concurrency, the serving engine and the latency/throughput targets you measure, not on the weight footprint alone.
KV Cache-aware Routing
The core differentiator of llm-d is intelligent routing that is aware of KV cache state.
This flow is a simplified example of configured EPP plugins. A Pod with cached prefixes may still lose selection because of load.
Figure 2. How the EPP selects an endpoint for a repeated-prefix request versus a new-prefix request
Routing Operation Principles
- Request reception: Client sends inference request to Inference Gateway
- Prefix analysis: Gateway calls the EPP through ext-proc; configured EPP plugins analyze the prefix
- Cache lookup: EPP evaluates candidate Pods using the configured cache index and metrics
- Intelligent routing: EPP combines cache and load scores to select an endpoint; Gateway proxies to that Pod
- Response return: vLLM returns inference results to client via Gateway
KV Cache-aware Routing Effects
| Metric | Cache miss (no prefix reuse) | Cache Hit (llm-d) | Improvement |
|---|---|---|---|
| TTFT (Time To First Token) | High (full prefill required) | Low (prefill skipped) | Reduced on prefix reuse (benchmark-dependent) |
| GPU Computation | Full prompt processing | Only new tokens processed | Saves recompute on reused prefix |
| Throughput | Baseline | Improved | Improves with prefix-reuse ratio (benchmark-dependent) |
KV Cache-aware routing is most effective in applications using identical system prompts. For example, in RAG pipelines that repeatedly reference the same context documents, reusing the prefix's KV cache can significantly reduce TTFT.
EKS Auto Mode Integration
Auto Mode Advantages and Limitations
Advantages:
- Automatic GPU driver management: AWS installs and updates the NVIDIA host driver and the NVIDIA container toolkit on Auto Mode GPU nodes
- Automatic NodeClass selection: Using
defaultNodeClass lets Auto Mode auto-select optimal AMI and driver version - Operational simplification: Removes the burden of installing and version-managing the host driver. It does not remove the need to verify that each application image's CUDA/runtime (the CUDA userspace shipped inside the container, e.g. the vLLM image) is compatible with the AWS-managed host driver and the target GPU architecture
- GPU Operator installable: Because AWS manages the NVIDIA driver, container toolkit and device plugin on Auto Mode nodes, install the GPU Operator with
driver.enabled=falseandtoolkit.enabled=falseand keep its device plugin off those nodes with the labelnvidia.com/gpu.deploy.device-plugin: "false"(see Transition Notes below); DCGM Exporter, NFD and GFD operate normally
Limitations:
- MIG/Time-Slicing restrictions: Users cannot change the NVIDIA device plugin configuration managed by Auto Mode. This does not mean that NodeClass itself is read-only.
- Custom AMI not available: Cannot pin a specific host driver or CUDA driver version; applications needing a pinned driver must use a node type where you own the driver (Managed Node Group or self-managed nodes)
Auto Mode vs Karpenter + GPU Operator Comparison
Auto Mode is suitable for large model serving without GPU driver management burden, while Karpenter is advantageous for workloads requiring advanced GPU features like MIG/Time-Slicing.
Detailed comparison and cost analysis: See EKS GPU Node Strategy — Node Type Comparison
GPU Instance Specifications
- p5e.48xlarge (H200): 100B+ parameter models, maximum memory utilization
- p5.48xlarge (H100): 70B+ parameter models, highest performance
- g6e family (L40S): 13B-70B models, cost-efficient inference
When llm-d ModelService requests GPUs via DRA (ResourceClaim), support depends on the EKS Kubernetes version and how nodes are provisioned. Per the EKS device management guide, DRA is a Kubernetes feature that reached general availability in Kubernetes 1.34.
- Supported node paths: AWS documents DRA as available on EKS 1.33+ (recommended on 1.34+), and only with static capacity provisioning in Karpenter (a static NodePool), EKS Managed Node Groups, or self-managed nodes running a DRA driver.
- Not supported: DRA is not currently compatible with EKS Auto Mode, and not compatible with Karpenter when capacity is dynamically provisioned. For DRA workloads under Auto Mode, use Managed Node Groups (or self-managed nodes / a static Karpenter NodePool) instead.
- Driver coexistence: A DRA driver and the matching device plugin for the same device type must not run on the same node simultaneously.
Details: EKS GPU Node Strategy — MNG Hybrid for DRA Workloads
llm-d v0.8.1 Key Features
Status follows the v0.8.1 release and documentation at that tag. Deployment patterns, implemented features, and API stability describe different things.
Table 2. llm-d v0.8.1 key features and their status at the reviewed baseline
| Feature | Description | Status at the reviewed baseline |
|---|---|---|
| Prefill/Decode Disaggregation | Deploy separate Prefill and Decode Pod groups with KV transfer | Well-lit path |
| Expert Parallelism (Wide EP) | Distribute experts for supported MoE model and hardware combinations | Well-lit path |
| LoRA-aware scheduling | lora-affinity-scorer considers loaded adapters and loading capacity; dynamic loading depends on model-server configuration | Implemented plugin, not a blanket GA hot-swap guarantee |
| Multi-model routing | HTTPRoute paths, headers, and weights select per-model InferencePools; routing on the JSON model body requires separate implementation/configuration | Gateway Mode pattern; distinguish the IPP proposal from default behavior |
| Gateway API Inference Extension | InferencePool inference.networking.k8s.io/v1; optional request policy uses llm-d InferenceObjective llm-d.ai/v1alpha2 | GIE v1 API / llm-d alpha API |
| Flow control | Integer InferenceObjective priority orders request queues | Graduated to production in the release; the configuration guide requires explicitly enabling the flowControl feature gate |
InferenceModel is the historical name of the earlier policy API. These schemas do not define model images, GPU allocation, or replica counts, and priority does not reserve GPUs.
Disaggregated Serving Concept
Disaggregated Serving separates the two phases of LLM inference for independent optimization:
Figure 3. Prefill Pod group and Decode Pod group connected through NIXL KV transfer
Table 3. Computational characteristics and typical tuning of the Prefill and Decode phases
| Phase | Characteristics | Typical tuning (workload-specific) |
|---|---|---|
| Prefill | Processes entire prompt at once (compute-bound) | Parallelism chosen to meet prefill latency; the llm-d example uses prefill TP=1 |
| Decode | Autoregressive token-by-token generation (memory-bound) | Parallelism chosen for KV-cache capacity and decode throughput; the llm-d example uses decode TP=5 |
TP values are not fixed by the stage role — the llm-d P/D disaggregation example runs prefill at TP=1 and decode at TP=5. Choose TP per model, hardware and measured targets.
NIXL (NVIDIA Inference Xfer Library): A common KV transfer library used by projects including Dynamo, llm-d, production-stack, and aibrix. NIXL moves KV blocks over a UCX-backed transport; it supports TCP, but high-bandwidth networking (InfiniBand, RoCE, or EFA) is recommended for production. The achievable transfer rate and overhead depend on the actual physical interconnect and topology, not on NIXL alone, and the TCP path remains a working fallback.
Disaggregated Serving on EKS Auto Mode
Allocate GPUs through Pod nvidia.com/gpu requests, as in the AWS GPU workload example. Auto Mode does not support MIG, but this does not require Prefill and Decode to run on different nodes. Pods can request different whole GPUs on the same multi-GPU node; placement depends on available GPUs, memory, and Pod scheduling constraints.
Separate NodePools are a design option when hardware or scaling policies need isolation. The counts below are illustrative, not measured performance results.
Prefill NodePool: TP=4, nvidia.com/gpu: 4 per Pod
Decode NodePool: TP=2, nvidia.com/gpu: 2 per Pod
Configure replicas, GPU requests, and nodeSelector/affinity in the model-server workload; use InferencePool to select those Pods. InferenceObjective request priority does not replace node placement or GPU isolation.
llm-d vs NVIDIA Dynamo
llm-d and NVIDIA Dynamo both provide LLM inference routing/scheduling but with different approaches. For detailed comparison, see NVIDIA GPU Stack — llm-d vs Dynamo Selection Guide.
Table 4. Comparison of llm-d and NVIDIA Dynamo across architecture, routing, and operations
| Item | llm-d | NVIDIA Dynamo |
|---|---|---|
| Governance | CNCF sandbox project (Apache 2.0), founded by Red Hat, Google Cloud, IBM Research, CoreWeave and NVIDIA | NVIDIA (Apache 2.0) |
| Architecture | Aggregated + Disaggregated | Aggregated + Disaggregated (equal support) |
| KV Cache Transfer | NIXL (TCP supported; IB/RoCE/EFA recommended) | NIXL (TCP supported; IB/RoCE/EFA recommended) |
| KV Cache Indexing | Prefix-aware routing | Flash Indexer (radix tree-based) |
| Routing | Gateway API + Envoy EPP | Dynamo Router + custom EPP (Gateway API integration) |
| Pod Scheduling | K8s default scheduler | KAI Scheduler (GPU-aware Pod placement) |
| Autoscaling | HPA/KEDA integration | Planner (SLO-based: profiling → autoscale) + KEDA/HPA |
| GPU Operator Required | Optional (Auto Mode compatible) | Dynamo Platform prerequisite; KAI Scheduler is an optional multi-node component |
| Complexity | Low | High |
| Strengths | K8s native, lightweight, fast adoption | Flash Indexer, KAI Scheduler, Planner SLO autoscaling |
- EKS Auto Mode + quick start: llm-d (GPU Operator optional)
- K8s-native, lightweight stacks: llm-d
- Maximum throughput at scale: Dynamo (Flash Indexer + Planner)
- Long context with multi-tier KV cache offload (GPU→CPU→SSD): Dynamo exposes this via KVBM; evaluate against your context length and latency targets
- K8s Gateway API standard compliance: llm-d
llm-d and Dynamo are complementary. They are independent parallel stacks that share NIXL (the KV transfer library) and the Gateway API; llm-d uses NIXL for KV cache transfer. Starting with llm-d and moving to Dynamo as requirements grow is practical, but a given GPU count does not by itself require migration — compare the configured features and your measured requirements.
Migration Path
Figure 4. Topology contrast between the Auto Mode + llm-d and Karpenter + Dynamo configurations
Phased transition path (optional designs, not a required ladder):
These phases are example configurations, not an ordered requirement; a larger GPU count does not by itself mandate a later phase or establish maximum performance. Choose a configuration from your workload's isolation, scheduling and DRA needs.
Table 5. Configuration and example fit for each phased transition path
| Phase | Configuration | Example fit |
|---|---|---|
| Phase 1 | Auto Mode + llm-d | PoC, dev environments |
| Phase 1.5 | Auto Mode + GPU Operator + llm-d | Added monitoring/scheduling operands |
| Phase 2a | Karpenter + llm-d Disaggregated | Mid-scale production, MIG utilization |
| Phase 2b | MNG + DRA + llm-d | DRA-required environments |
| Phase 3 | Karpenter + Dynamo | Dynamo-specific features (Flash Indexer, Planner) |
Auto Mode and self-managed Karpenter can coexist in the same cluster. In Phase 1.5, keep the GPU Operator's device plugin off the AWS-managed Auto Mode nodes so it does not conflict with the AWS-managed device plugin. Set the label nvidia.com/gpu.deploy.device-plugin: "false" in the Auto Mode NodePool template, as in the configuration reviewed in awslabs/ai-on-eks PR #288. The GPU Operator applies its nvidia.com/gpu.deploy.* labels only where a label is absent or empty, so it keeps an existing false value and skips that operand on those nodes (addGPUStateLabels). The ClusterPolicy has no per-component or common DaemonSet nodeSelector/affinity field for this. Setting devicePlugin.enabled=false instead turns the device plugin off on every node, so use it only when no node in the cluster relies on the GPU Operator's device plugin.
Monitoring
Key Monitoring Metrics
| Metric | Description | Normal Range |
|---|---|---|
| vllm:num_requests_running | Requests in execution batches (Gauge) | Varies by workload |
| vllm:num_requests_waiting | Requests waiting to be scheduled (Gauge) | Workload SLO choice (sustained high values signal capacity pressure) |
| vllm:kv_cache_usage_perc | KV cache usage as a 0–1 ratio (1 = 100%) | Workload SLO choice (e.g. ~0.9 sustained raises preemption/queueing) |
| rate(vllm:generation_tokens_total[1m]) | Per-second rate of the generation-token counter | Varies by model/GPU |
| rate(vllm:prompt_tokens_total[1m]) | Per-second rate of the prompt-token counter | Varies by model/GPU |
| histogram_quantile(0.95, vllm:e2e_request_latency_seconds_bucket) | P95 of the e2e request-latency histogram (seconds) | Workload SLO choice (e.g. P95 < 30s) |
Model Loading Time
| Loading Method | Relative Speed | Notes |
|---|---|---|
| HuggingFace Hub (initial) | Slowest | Full weight download + load; scales with model size and internet bandwidth |
| S3 Cache (same region) | Faster | Loads from same-region S3, skipping internet download; depends on S3 throughput |
| Node Local Cache | Fastest | Weights already on local disk when redeploying on the same node; only local disk I/O and GPU load remain |
Cost Optimization
| Strategy | Description | Estimated Savings |
|---|---|---|
| Savings Plans | 1-year/3-year Compute Savings Plans commitment (discount varies by term and upfront option) | Discount vs On-Demand (per commitment terms) |
| Off-Peak Scale Down | Reduce replicas during nights/weekends (CronJob); saves in proportion to billed instance-hours removed | Scales with removed instance-hours |
| Model Quantization | May reduce GPU count via INT8/INT4 (validate accuracy and throughput impact) | Only when billed GPU count actually drops |
| Spot Instances | Apply to fault-tolerant workloads (risk of interruption) | Spot discount vs On-Demand (variable, with interruption risk) |
| TP Optimization | Use minimum TP value appropriate for model size | Avoids over-allocating GPUs |
As an example rate, p5.48xlarge On-Demand in us-west-2 is about $55.04/hr (reflecting the June 2025 EC2 GPU price reduction — the announcement states up to a 45% reduction for P5 rather than a specific rate, so confirm the current figure on the EC2 pricing page). Two instances therefore cost about $55.04 × 2 × 720h = $79,258/month (720h) to $55.04 × 2 × 730h = $80,360/month (730h). This covers the EC2 instances only; the EKS Auto Mode management fee, EBS volumes, data transfer and other charges are additional. Always clean up resources after testing.
EKS Auto Mode GPU Instance Support Status (Verified 2026-04)
Instance Support Matrix
Table 6. EKS Auto Mode GPU instance support status and observations by instance type
| Instance Type | GPU | VRAM (Total) | Auto Mode Support | Observation |
|---|---|---|---|---|
| g5.xlarge~48xlarge | A10G | 24~192GB | Supported | Provisioning observed |
| g6.xlarge~48xlarge | L4 | 24~192GB | Supported | Provisioning observed |
| g6e.xlarge~48xlarge | L40S | 48~384GB | Supported | Provisioning observed |
| p4d.24xlarge | A100 40GB x 8 | 320GB | Supported | Dry-run only (API admission validation, not a provisioning guarantee) |
| p5.48xlarge | H100 80GB x 8 | 640GB | Supported | Spot provisioning observed (us-east-2) |
| p5en.48xlarge | H200 141GB x 8 | 1,128GB | Supported | Dry-run passes; capacity/offering matching may still fail |
| p6-b200.48xlarge | B200 180GB x 8 | 1,440GB | Supported (since 2026-04-10) | Specify eks.amazonaws.com/instance-family: p6-b200 in NodePool requirements; regional B200 and Capacity Block availability may limit provisioning |
EKS Auto Mode's managed Karpenter supports p6-b200.48xlarge as of 2026-04-10. "Dry-run" entries above reflect that the Kubernetes dry-run admission path validated the request; dry-run validates API admission and does not guarantee that capacity can actually be provisioned. Regional B200 capacity and Capacity Block availability can still limit real provisioning.
Per-Region GPU Capacity Availability
Table 7. Per-region p5.48xlarge On-Demand/Spot availability and observed price
| Region | p5.48xlarge On-Demand | p5.48xlarge Spot | Spot Price |
|---|---|---|---|
| ap-northeast-2 (Seoul) | InsufficientCapacity (unverified) | Unconfirmed | -- |
| us-east-2 (Ohio) | Availability varies | Acquired (unverified) | $13-15/hr (observed) |
Spot Price Comparison (us-east-2, 2026-04): These are point-in-time observations without stored artifacts; treat them as unverified and expect no future availability guarantee. Against the $55.04/hr On-Demand example, an observed $13-15/hr Spot price implies about (55.04-15)/55.04 = 72.75% to (55.04-13)/55.04 = 76.38% savings — roughly 73-76%, not a 90% upper bound. For detailed pricing, see GPU Resource Management — Cost Optimization Strategies.
GPU Quota Notes
Table 8. GPU instance Service Quotas and example account-level values
| Quota Name | Applicable Instances | AWS Default | Applied Value Example (one account) |
|---|---|---|---|
| Running On-Demand P instances | p4d, p4de, p5, p5en | 0 vCPU | 384 (EC2 raises quotas automatically with usage) |
| Running On-Demand G and VT instances | g5, g6, g6e | 0 vCPU | 64 (EC2 raises quotas automatically with usage) |
When setting instance-category: [g, p] together in GPU NodePool, Karpenter may try G-type instances first. To use P-type only, explicitly specify instance-category: [p].
Next Steps
- EKS GPU Node Strategy -- Auto Mode vs Karpenter vs Hybrid Node, per-model-size cost analysis
- vLLM Model Serving and Performance Optimization -- vLLM basics and deployment
- MoE Model Serving Guide -- Mixture of Experts model serving
- GPU Resource Management -- GPU cluster resource management