AgentCore and EKS: Runtime, Model and Gateway Evaluation Plan
An unexecuted plan to compare runtime hosting, model serving and gateways under controlled quality, isolation, performance and cost requirements
6 docs tagged with "llm-d"
View all tagsDisaggregated Serving + LWS Multi-Node
Prefill/Decode separation architecture and NIXL common KV transfer engine, LeaderWorkerSet-based 700B+ large MoE model multi-node deployment guide
Inference Frameworks
vLLM·llm-d·MoE·NeMo — AI framework layer for actual model serving, distributed inference, and fine-tuning on GPUs
KV Cache Optimization (vLLM Deep Dive + Cache-Aware Routing)
Summary of core technologies like vLLM PagedAttention, Continuous Batching, FP8 KV Cache, and comparison of llm-d/NVIDIA Dynamo KV Cache-Aware Routing and Gateway configuration
llm-d Based EKS Distributed Inference Guide
llm-d architecture concepts, KV Cache-aware routing, Disaggregated Serving, EKS Auto Mode integration strategy
Model Serving & Inference Infrastructure
A guide to the GPU infrastructure, inference framework, and inference optimization layers, with a single map of the end-to-end LLM inference request path and per-layer tuning levers — inference gateway, prefill/decode disaggregation, KV cache-aware routing, LMCache, and cache-hit strategy.