Inference Optimization on EKS
Overview
Guides for measuring inference latency, throughput, GPU memory, and cost, then choosing an optimization that addresses the observed bottleneck. Start with the L0–L5 tuning view in the inference infrastructure overview. These tuning layers are separate from the platform's six runtime layers.
Results depend on the model, input and output lengths, concurrency, hardware, and routing policy. This page helps select detailed guides; it does not prescribe a configuration that guarantees production performance or savings.
Covered Content
All documents in this category are listed below. Gateway configuration and model selection policies belong to the separate inference routing category.
KV Cache Optimization
Summary of core technologies like vLLM PagedAttention, Continuous Batching, FP8 KV Cache, and comparison of llm-d/NVIDIA Dynamo KV Cache-Aware Routing and Gateway configurationDisaggregated Serving
Prefill/Decode separation architecture and NIXL common KV transfer engine, LeaderWorkerSet-based 700B+ large MoE model multi-node deployment guideLMCache
The concept of LMCache — offloading KV cache beyond GPU memory to CPU and disk and sharing it across inference instances — and its relationship to vLLM prefix cache, NIXL, and kvaware routing.Cache-Hit Strategy
Unifying the three layers of inference caching (KV/prefix, exact-response, semantic) into a single decision framework with hit-rate targets, measurement points, and tuning levers for each layer.Semantic Caching
LLM Gateway-level semantic caching strategy and implementation options comparison (GPTCache, Redis Semantic Cache, Portkey, Helicone, Bifrost+Redis)Tiered Gateway Architecture
Single definition of the Agentic AI Platform gateway layers: Tier 1 Ingress, Tier 2 Inference Routing (Inference Extension) and LLM API Gateway, and the Agent Data Plane — their role separation and how to fill each layerGateway Routing Strategy
kgateway + Bifrost/LiteLLM 2-Tier architecture with Cascade Routing, Semantic Router, and Hybrid Routing design patternsRequest Cascading
Complexity-based automatic model routing — comparison of LLM Classifier, LiteLLM, and vLLM Semantic Router approaches, RouteLLM research reference, and cost savingsCascade Routing Tuning
Guide to tuning Inference Gateway Cascade Routing classification thresholds, Canary rollout, Fallback, and cost drift alerts based on production tracesOpenClaw AI Gateway
Deploy OpenClaw AI Agent Gateway on EKS with cost optimization, and achieve full observability using LiteLLM Auto Router + Cilium Hubble + LangfuseAutoscaling & Deployment Ops
2-Tier GPU autoscaling (KEDA·Karpenter), DRA compatibility, and operational lessons learned from large MoE model (GLM-5·Kimi K2.5) deployments for LLM servingKey Topics by Document
- Cache reuse: KV cache optimization → LMCache → cache-hit strategy
- Latency and throughput isolation: compare the benefits and transfer costs of prefill/decode separation in disaggregated serving.
- Response-reuse quality: review cache keys, tenant boundaries, and incorrect-hit evaluation in semantic caching.
- Capacity and recovery: review queues, cold starts, and deployment failures in GPU autoscaling.
Key Performance Metrics
Compare before and after against the service SLO and the same request set. There is no universal target for the metrics below.
Table 1. What to inspect and conditions to record for each inference performance metric
| Metric | What to inspect | Conditions to record |
|---|---|---|
| TTFT | Request-start to first-token latency distribution | Queue-time inclusion, input length, concurrency |
| Output-token throughput | Generated tokens in completed requests / measurement duration | Per-client rate versus aggregate server throughput |
| GPU memory and utilization | Memory headroom, compute and memory bottlenecks | GPU type, precision, batch settings |
| Cache-hit ratio | Hits and lookups for each reuse unit | Token or request denominator, cache type, cold/warm interval |
| Tail latency and errors | p95/p99 response latency and completion rate | Timeouts, retries, and failed requests |
| Quality and cost | Evaluation pass rate and total cost per successful request | Evaluation set, idle GPU cost, gateway and storage costs |
EKS GPU Infrastructure Strategy
Three Deployment Model Comparison
Separate node management from ownership of GPU software. GPU count or model size alone does not determine the node-management approach.
Table 2. Management boundary and detailed checks by EKS GPU deployment model
| Approach | Management boundary | Detailed checks |
|---|---|---|
| EKS Auto Mode | EKS manages the GPU driver and device plugin | Available features and support for user-installed components |
| Karpenter | Provisions nodes according to NodePool policy | Selected AMI, GPU components, static versus dynamic capacity |
| Managed node group | Manages capacity through a node group and AMI | Pre-installed components versus separately installed drivers/plugins |
Check DRA support against both Kubernetes version and capacity provisioning. The current EKS NVIDIA device guide distinguishes Karpenter static capacity, managed node groups, and self-managed nodes while excluding Auto Mode. See GPU node strategy for selection criteria.
GPU Instance Selection Matrix
Table 3. Evaluation criteria for each GPU instance selection decision
| Decision | Evaluation criteria |
|---|---|
| Memory | Weight precision, KV cache, concurrency, runtime headroom |
| Communication | Bandwidth for single-GPU, within-node, or cross-node parallelism |
| Availability | Capacity, quotas, and alternatives in the target Region and AZs |
| Cost | Quote date, Region, purchase option, and measured throughput |
Do not derive GPU count from parameter count alone. Review memory settings in vLLM serving and allocation policies in GPU resource management.
Auto Mode GPU Operator Hybrid Configuration
A separate GPU Operator must not take duplicate ownership of components managed by Auto Mode. This overview does not provide a generic installation procedure that applies AL2023 or Bottlerocket GPU Operator options to Auto Mode.
The accelerated AMI guide describes options according to each AMI's pre-installed components. Even when only observability is needed, verify support on the target nodes and consult the component descriptions in the NVIDIA GPU stack.
Recommended Architecture by Model Scale
Decision Flow
Figure 1. Inference deployment selection flow by model scale
3-Tier Recommended Configuration
These are evaluation stages, not fixed specifications tied to parameter counts.
Table 4. Condition and next guide for each 3-Tier deployment stage
| Stage | Condition | Next guide |
|---|---|---|
| Single GPU | Meets both memory and SLO requirements | vLLM serving |
| Within-node parallelism | A single GPU reaches a memory or performance limit | MoE serving |
| Multi-node or disaggregated serving | Measurements show within-node capacity is insufficient | Disaggregated serving |
Hybrid Architecture: Complete Picture
When combining EKS, on-premises infrastructure, and managed model APIs, use the tiered gateway guide to identify each layer's responsibilities. Validate fallback API and tool-call compatibility, data-transfer policy, quotas, and response quality. A routing path alone does not guarantee uninterrupted service.
Migration Path
- Record performance, quality, and cost on a single baseline path.
- Change one cache, routing, or batching setting for an observed bottleneck.
- Check regressions with the same evaluation set and load conditions.
- Validate limited traffic before expansion and define when to restore the original path.
References
Official Documentation
Papers & Technical Blogs
Check implementation projects against the selected release and its documented examples.
Related Documentation
- Serving optimization monitoring — measurement units and query validation
- Prefix-cache tuning and accuracy — experiment design and quality criteria
- Routing strategy — model selection and fallback boundaries