8 docs tagged with "optimization"

Disaggregated Serving + LWS Multi-Node

Prefill/Decode separation architecture and NIXL common KV transfer engine, LeaderWorkerSet-based 700B+ large MoE model multi-node deployment guide

EKS Pod Resource Optimization Guide

CPU/Memory resource configuration, QoS classes, VPA/HPA autoscaling, and resource right-sizing strategies for Kubernetes Pods

GPU Autoscaling & Large Model Deployment Operations

2-Tier GPU autoscaling (KEDA·Karpenter), DRA compatibility, and operational lessons learned from large MoE model (GLM-5·Kimi K2.5) deployments for LLM serving

Inference Optimization on EKS

EKS architecture overview for maximizing LLM Inference performance — starting point for vLLM, KV Cache-Aware Routing, Disaggregated Serving, LWS multi-node, and GPU autoscaling

KV Cache Optimization (vLLM Deep Dive + Cache-Aware Routing)

Summary of core technologies like vLLM PagedAttention, Continuous Batching, FP8 KV Cache, and comparison of llm-d/NVIDIA Dynamo KV Cache-Aware Routing and Gateway configuration

Large-Scale EKS Cost Management: Allocation, Optimization, and Verification

FinOps guidance for Amazon EKS cost allocation and optimization with SCAD, CUR 2.0, Karpenter v1.13, tagging, per-container rightsizing, and ROI verification.

MCP Tool Token Optimization Patterns

Token optimization patterns for MCP-based agents. Explains the upfront loading cost that follows from the host's tool-injection strategy and four techniques — Progressive Discovery, a tool compression proxy, Code Execution and prompt cache alignment — with published savings figures and the conditions they were measured under.

vLLM Model Serving

vLLM PagedAttention, parallelization strategies, Multi-LoRA, and hardware support architecture