Skip to main content

Inference Gateway & LLM Gateway Routing Strategy

79 min read

This document covers design principles for 2-Tier gateway architecture and routing strategies (Cascade / Semantic Router / Hybrid). For actual deployment procedures including Helm installation, HTTPRoute manifests, and OTel integration, refer to Inference Gateway Deployment Guide.

Overview​

In large-scale AI model serving environments, distinguishing infrastructure traffic management from LLM provider abstraction helps clarify responsibilities. The 2-Tier layout below is a design option for separating ownership, policies, and scaling; it is not a requirement for two traffic gateways in every llm-d deployment.

2-Tier Gateway Architecture:

  • L1 (Ingress Gateway): kgateway — Kubernetes Gateway API standard, traffic routing, mTLS, rate limiting
  • L2-A (Inference Gateway): Bifrost/LiteLLM — Provider integration, cascade routing, semantic caching
  • L2-B (Data Plane): agentgateway — MCP/A2A protocols, stateful session management

Each tier is managed independently, separating infrastructure and AI workloads.


2-Tier Gateway Architecture​

Gateway layer definitions are unified in a dedicated document

The platform-wide gateway-layer terminology and role definitions are consolidated in Tiered Gateway Architecture. This document focuses on the routing strategy of Tier 2-A (LLM API Gateway). For in-cluster inference pod routing (Tier 2 ① Inference Extension), see the Gateway API Inference Extension section below.

Gateway Layer Separation​

LLM inference platforms must clearly distinguish 3 different Gateway roles. (For the full layer definitions, see Tiered Gateway Architecture)

Table 1. Gateway roles, implementations, and locations by type

Gateway TypeRoleImplementationLocation
Ingress GatewayExternal traffic ingress, TLS termination, path-based routingkgateway (NLB integration)Tier 1
LLM API GatewayModel selection, intelligent routing, request cascading (external/internal model abstraction)Bifrost / LiteLLMTier 2-A
Agent Data PlaneMCP/A2A protocols, stateful sessions, tool routingagentgatewayTier 2-B

Terminology note: Here, Tier 2-A "LLM API Gateway" is a provider proxy (Bifrost/LiteLLM) that abstracts the model API. It serves a different purpose from the Gateway API Inference Extension (Tier 2 ①, later in this document) that routes to in-cluster inference pods.

Figure 1. Request distribution across the 2-Tier Gateway layers

Expand the diagram to read labels and follow connections.
Diagram

Responsibilities assigned in this design: (These are responsibilities this 2-Tier layout assigns to each component, not exclusive product capabilities. Not all three components are required.)

  • Ingress Gateway (kgateway): In this design, handles network-level traffic control only; model selection is delegated to the L2-A gateway.
  • LLM API Gateway (Bifrost/LiteLLM): Analyzes requests to select a model and tracks cost. Complexity-based selection is not automatic — it requires explicitly configuring a classifier/policy, such as LiteLLM's complexity_router (classifier_type: heuristic/llm/etc.) or Bifrost routing rules.
  • Agent Data Plane (agentgateway): In this design, handles AI-specific protocols (MCP/A2A) and maintains stateful sessions. agentgateway can itself route LLM providers and inference endpoints, but here that responsibility is kept at L2-A to separate roles.

Overall Architecture​

Figure 2. End-to-end 2-Tier Gateway architecture with observability links

Expand the diagram to read labels and follow connections.
Diagram

Responsibility Separation by Tier​

Table 2. Components, responsibilities, and protocols by tier

TierComponentResponsibilityProtocol
Tier 1 (Ingress Gateway)kgateway (Envoy-based)Traffic routing, mTLS, rate limiting, network policiesHTTP/HTTPS, gRPC
Tier 2-A (LLM API Gateway)Bifrost / LiteLLMIntelligent model selection, cost tracking, request cascading, semantic cachingOpenAI-compatible API
Tier 2-B (Agent Data Plane)agentgatewayMCP/A2A session management, self-hosted inference routing, Tool Poisoning preventionHTTP, JSON-RPC, MCP, A2A

Traffic Flow​

External LLM: Client → kgateway → Bifrost/LiteLLM (Cascade + Cache) → OpenAI → Response + Cost tracking Self-hosted vLLM: Client → kgateway → agentgateway → vLLM → Response


kgateway (L1 Inference Gateway)​

Gateway API-based Routing​

kgateway implements the Kubernetes Gateway API standard, enabling vendor-neutral configuration.

ComponentRoleDescription
GatewayClassGateway implementation definitionDesignate Kgateway controller
GatewayEntry point definitionConfigure listeners, TLS, addresses
HTTPRouteRouting rulesPath, header-based routing
BackendModel servicevLLM, TGI and other inference servers

Gateway API v1.6.0 was released on 2026-06-29, graduating UDPRoute and TCPRoute to GA (standard channel). kgateway's Gateway API support is pinned per release and is for the standard channel: kgateway 2.4.x (released 2026-07-23) supports Gateway API 1.4–1.6, 2.3.x (2026-05-19) supports 1.3–1.5, and 2.2.x/2.1.x support 1.2–1.4. So v1.6 is not "future" — it is already supported by kgateway 2.4.x. Before using a specific feature, confirm the exact kgateway version and Gateway API channel (standard/experimental) combination against the kgateway version support table.

Dynamic Routing Concepts​

Table 3. Criteria and use cases by dynamic routing type

Routing TypeCriteriaUse Case
Header-basedx-model-id, x-providerBackend selection by model/provider
Path-based/v1/chat/completions, /v1/embeddingsService separation by API type
Weight-basedbackendRef weightCanary deployment, A/B testing
Composite conditionsHeaders + Path + TierPremium/standard customer backends

Canary deployments start with 5-10% traffic and gradually increase, with immediate rollback via weight=0 on issues.

Load Balancing Strategies​

Table 4. Load balancing strategies and suitable scenarios

StrategyDescriptionSuitable Scenario
Least RequestRoutes to the backend with the fewest requests (default when no kgateway BackendConfigPolicy is set)General workloads with uneven processing times
Round RobinSequential distributionUniform model instances
RandomRandom distributionLarge backend pools
Consistent HashSame key → Same backendKV Cache reuse, session affinity

When no load balancing algorithm is set in a kgateway BackendConfigPolicy, the default is least request (pick the instance with the fewest requests), not round robin.

Consistent Hash can help with prefix cache reuse in LLM inference, but the TTFT benefit is conditional, not guaranteed. Hit rates improve only when the same user's requests actually share a prefix, that prefix is still resident (warm) on the target vLLM instance, and hash membership is stable. Adding/removing backends reshuffles the hash ring, and differing prefixes erase the benefit — so measure hit rate and TTFT on your actual workload to confirm.

Topology-Aware Routing (same-AZ preference)​

Prioritizing same-AZ Pod-to-Pod communication can reduce cross-AZ data transfer cost and latency. This is not applied automatically by a Kubernetes version alone; it depends on which mechanism you explicitly configure, and these are distinct layers:

  • Service trafficDistribution: PreferClose (GA in Kubernetes 1.31): a Service-level setting that hints kube-proxy/EndpointSlice to prefer same-zone endpoints. It supersedes the older service.kubernetes.io/topology-mode: Auto (topology-aware hints).
  • Gateway/EPP path: when a Gateway API data plane or EPP (Inference Extension) routes directly to Pod addresses, it may bypass kube-proxy, so Service-level topology hints do not apply as-is. On that path, locality must be controlled by the gateway's/scheduler's own locality or zone-aware settings.

So do not infer "locality is guaranteed because the cluster is at version X." State the actual data plane handling the traffic (kube-proxy Service path vs. gateway/EPP direct routing) and the topology mechanism configured there.

🚀 Topology-Aware Routing Effects
MetricBeforeTopology-AwareImprovement
Cross-AZ TrafficHighReduced (conditional)
Prioritizing same-AZ reduces cross-AZ transfer and cost (actual savings depend on traffic distribution/AZ layout; measure to confirm)
LatencyHigh (cross-AZ)Low (same AZ)
Same-AZ routing can lower P99 latency by cutting network hops (magnitude varies by workload/baseline; measure to confirm)
Network BandwidthLimitedOptimized (conditional)
Fewer cross-AZ hops can raise effective throughput (actual effect requires measurement)

Failure Handling Concepts​

Table 5. Failure-handling mechanisms and LLM inference considerations

MechanismDescriptionLLM Inference Considerations
TimeoutMaximum processing time per requestLLM long response generation takes tens of seconds. Adequate timeout needed (120s+)
RetryAuto-retry on 5xx, timeout, connection failureMax 3 retries. Infinite retries cause system overload
Circuit Breaker / outlier detectionTemporarily eject backends on passive health signals (e.g. consecutive failures)maxEjectionPercent caps the share of backends ejected at once; it does not guarantee a number of available backends. The panic threshold (default 50%) and active health checks also affect availability, so "at least half available" does not hold

Envoy outlier detection is passive health checking: it ejects hosts based on failures observed in request results. maxEjectionPercent only caps the share of hosts that may be ejected; it does not guarantee any specific number of available hosts, and actual availability also depends on active health checks and the panic threshold.

In Gateway API timeouts, backendRequest is the per-try timeout covering the entire backend response, and request is the total request time including retries. So backendRequest is not a "first byte" timeout — it covers the whole backend response for that attempt, which means a long SSE streaming response can be cut off by it. For streaming, set first-byte (TTFB), idle, and whole-stream timeouts distinctly as appropriate. POST retries require idempotency guarantees (caution with tool calls).


LLM Gateway Solution Comparison​

Major Solution Comparison Table​

Table 6. LLM Gateway solutions compared by features, cascade routing, and license

SolutionLanguageKey FeaturesCascade RoutingLicenseBest For
BifrostGoHigh-performance gateway; conditional routing rules (CEL) over complexity_tier/model/params/capacity signals; failoverRouting rules (complexity_tier) + external classifierApache 2.0High performance, low cost, self-hosted
LiteLLMPython100+ providers; multi-strategy routing (latency/usage/cost/least-busy) + native complexity_router (v1.101.0) + custom strategiesNative complexity_router (SIMPLE/MEDIUM/COMPLEX/REASONING) or custom strategyMITPython ecosystem, rapid prototyping
vLLM Semantic RouterStandalone service (Envoy ext_proc)A vLLM-project signal-driven router deployed standalone in front of OpenAI-compatible backends (not vLLM-only); embedding-similarity classificationEmbedding similarity-basedApache 2.0vLLM and other OpenAI-compatible backends
PortkeyTypeScriptSOC2 certified, semantic caching, Virtual KeysSupportedProprietary + OSSEnterprise, compliance
Kong AI GatewayLua/CMCP support, leverages existing Kong infra. Note: complexity/semantic routing via ai-proxy-advanced is a Kong Enterprise (paid) edition plugin and differs from the OSS ai-proxyPlugin (edition-dependent)Apache 2.0 (core) / Enterprise (advanced AI plugins)Existing Kong users
HeliconeRustGateway + Observability integrated, high performanceSupportedApache 2.0High performance + observability needed
OpenRouterSaaS (hosted)Unified API for 400+ models·60+ providers, provider fallback, OpenAI-compatibleProvider routing supportedSaaS (commercial)Fast multi-provider integration, prototyping
Why this design does not combine LiteLLM and Kong (a local simplification)

In this design we use only one of LiteLLM or Kong AI Gateway as the LLM API gateway (Tier 2-A). Their roles overlap, so standardizing on one is a design choice to reduce operational complexity — not a claim that combining them (e.g. Kong in front + LiteLLM behind) is technically impossible or prohibited. That combination simply adds hops, configuration, and observability points, and lacks a widely published verified reference. For selection criteria from the tenancy-model and budget-enforcement perspective, see AI Gateway Multi-Tenancy — Selection Criteria.

Connecting L1 Budget Signals to Fallback Routing

Among L1 signals, budget is directly tied to governance policy. When a tenant's budget is exhausted, the L1 gateway can block the request (hard budget) or take a fallback path that downgrades to a lower-cost model. For per-tenant budget hierarchies and enforcement, see AI Gateway Multi-Tenancy; for the block/alert/fallback policy matrix on budget overrun, see LLM FinOps Chargeback. These policy decisions all happen at L1; L2 (KV-aware Pod selection) does not handle budget signals.

Self-hosted vs SaaS

In the table above, Bifrost·LiteLLM·Helicone·vLLM Semantic Router are self-hosted (deployed in-cluster), whereas OpenRouter is hosted SaaS. SaaS provides instant access to 400+ models and delegates provider fallback/billing, which is advantageous for fast integration and prototyping. However, since prompts are sent to an external service, review the governance considerations for environments with data sovereignty or regulatory requirements.

Bifrost vs LiteLLM​

Bifrost: A Go-based high-performance gateway offering lower memory use and higher throughput than Python (public benchmark: ~11 µs (<100 µs) overhead @ 5k RPS; "50x faster than LiteLLM" is a tagline summarizing the vendor's own head-to-head benchmark — public, but not third-party verified — reporting 54x P99, 40x overhead, 9.5x throughput; distinguish latency vs overhead vs throughput and do not infer performance or security from language alone). Beyond performance, supply-chain/security posture can also factor in: a single static Go binary has a smaller runtime dependency surface than a broad Python dependency tree, though actual risk varies by version/configuration and should be checked against each project's security advisories at adoption time. Conditional routing rules are CEL expressions that can read model, headers, params, capacity metrics (budget_used/tokens_used/request), and an auto-classified complexity_tier — only "SIMPLE", "MEDIUM", or "COMPLEX" (there is no REASONING tier) computed by embedding the prompt and matching the nearest reference phrase; when classification is unavailable, complexity_tier is treated as unknown and the rule does not match. Helm Chart deployment, OpenAI-compatible API. (Bifrost routing rules)

LiteLLM: 100+ provider support; multi-strategy routing (simple-shuffle·latency-based-routing·usage-based-routing-v2·least-busy·cost-based-routing) and custom strategies via CustomRoutingStrategyBase. As of v1.101.0, complexity-based routing is provided by a native complexity_router (auto-router) with built-in tiers SIMPLE/MEDIUM/COMPLEX/REASONING and a configurable classifier_type (heuristic·llm·heuristic_first·hybrid·custom·heuristic_v2). It is not enabled by a one-line routing_strategy: complexity-based toggle (that string does not exist); you configure a complexity_router block. For Langfuse, see the "Langfuse Integration" section (the one-line success_callback: ["langfuse"] is the SDK-callback path). Direct LangChain/LlamaIndex integration. However, being Python-based, it has lower throughput and higher memory use than Go. (LiteLLM routing, complexity_router v1.101.0)

Selection Criteria​

Table 7. Recommended solution and rationale by use case

Use CaseRecommended SolutionReason
Intelligent cascade (convenience priority)LiteLLMNative complexity_router + 100+ providers, Python ecosystem
Intelligent cascade (performance priority)BifrostRouting rules (complexity_tier) + external classifier, low Go overhead
vLLM standalone environmentvLLM Semantic RoutervLLM-project router (standalone ext-proc service), lightweight classification
High performance, low cost self-hostedBifrostGo-based low overhead (~tens of µs), low memory
Python ecosystem (LangChain)LiteLLMNative integration, 100+ providers
Enterprise compliancePortkeySOC2/HIPAA/GDPR, Semantic Cache
High performance + observability integratedHeliconeRust-based All-in-one

Table 8. Recommended gateway stacks by operational scenario

ScenarioRecommended StackReason
Startup/PoCkgateway + LiteLLMLow cost, 100+ provider integration. Deployment time depends on environment and cluster readiness (a simple PoC can be up in tens of minutes)
Self-hosted focused (performance)kgateway + Bifrost (CEL routing rules) + agentgatewayHigh performance, external+self-hosted pool 2-Tier
Enterprise multi-providerkgateway + Portkey + LangfuseMany-provider integration and observability. Compliance is not achieved by integration alone — verify Portkey's certification (e.g. SOC2) scope and your own operational/audit requirements separately
Hybrid (external+self-hosted)kgateway + Bifrost/LiteLLM + agentgatewayExternal via Bifrost/LiteLLM, self-hosted via agentgateway
Global deploymentCloudflare AI Gateway + kgatewayEdge caching, DDoS protection

These "recommendations" summarize typical fit for each stack. Provider counts and certification scope change by product, edition, and date, so confirm them against each vendor's docs at adoption time, and judge "deployed" by acceptance criteria (health checks passing, end-to-end request success) rather than an "n-minute deployment" figure.


Request Cascading: Intelligent Model Routing​

Request Cascading is an intelligent optimization technique that automatically analyzes request complexity and routes to the appropriate model. The three patterns (weight-based, fallback-based, intelligent routing), implementation approach comparison (LLM Classifier, LiteLLM, vLLM Semantic Router), RouteLLM research reference, and cost savings are covered in detail in Request Cascading — Intelligent Model Routing.

For threshold/keyword tuning and misroute detection operations, see Cascade Routing Tuning.


Gateway API Inference Extension​

Kubernetes Gateway API enables managing LLM inference as Kubernetes-native resources through Inference Extension.

Core CRDs (Custom Resource Definitions)​

The reviewed baseline is llm-d v0.8.1 (2026-06-26), router v0.9.0, GIE v1.5.0, and Gateway API v1.5.1. Primary schemas: llm-d v0.8.1, InferencePool v1, InferenceObjective v1alpha2, HTTPRoute v1.

Table 9. Owner, responsibility, and example of each Inference Extension CRD

ResourceOwner / APIResponsibilityExample
Deployment / LeaderWorkerSetKubernetes / separate workload APIImages, model arguments, Pod GPU requests, replica countsWorkload replicas: 3 and nvidia.com/gpu
InferencePoolGIE, inference.networking.k8s.io/v1Select same-namespace Pods and define ports/EPP referenceselector.matchLabels, targetPorts, endpointPickerRef
InferenceObjectivellm-d, llm-d.ai/v1alpha2 (alpha, optional)Request priority for a specific poolpoolRef, integer priority: 10
HTTPRouteGateway API, gateway.networking.k8s.io/v1Select pools by path, header, or weightbackendRefs.kind: InferencePool

InferenceModel is the historical name of the earlier policy API. This baseline uses llm-d InferenceObjective. criticality: high, model images, GPU requests, and replica counts are not InferenceObjective/InferencePool fields. Distinguish the GIE v1 InferencePool API from the llm-d alpha policy API.

There is no LLMRoute kind in the reviewed GIE or llm-d Router CRDs. Even the router's llmroute.yaml test file uses kind: HTTPRoute. Do not treat a filename as an API kind or copy that test file's old API group.

Gateway API Inference Extension Integration​

In Gateway Mode, one compatible Gateway can handle both traditional Services and InferencePools. The EPP returns its selection to the Gateway and does not proxy inference traffic itself. A separate edge Gateway is an option when retaining existing ingress or separating security/ownership; account for extra hops, operational cost, and consistent timeout, retry, streaming, and authentication-header handling.

Solid lines show requests/responses and EPP calls; dotted lines show configuration references.

Figure 3. Resource and EPP wiring of the Gateway API Inference Extension

Expand the diagram to read labels and follow connections.
Diagram

This example defines routing resources only. It requires an existing llm-d namespace, inference-gateway Gateway, inference-epp Service (EPP on port 9002 configured for gpu-pool), and model-server Pods labeled app: vllm listening on port 8000. Install Gateway API v1.5.1, the GIE v1.5.0 InferencePool CRD, the router v0.9.0 InferenceObjective CRD, and compatible controllers.

apiVersion: inference.networking.k8s.io/v1
kind: InferencePool
metadata:
name: gpu-pool
namespace: llm-d
spec:
selector:
matchLabels:
app: vllm
targetPorts:
- number: 8000
endpointPickerRef:
name: inference-epp
kind: Service
port:
number: 9002
failureMode: FailClose
---
apiVersion: llm-d.ai/v1alpha2
kind: InferenceObjective
metadata:
name: interactive
namespace: llm-d
spec:
poolRef:
group: inference.networking.k8s.io
kind: InferencePool
name: gpu-pool
priority: 10
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: inference-route
namespace: llm-d
spec:
parentRefs:
- name: inference-gateway
rules:
- matches:
- path:
type: PathPrefix
value: /v1
backendRefs:
- group: inference.networking.k8s.io
kind: InferencePool
name: gpu-pool
port: 8000

priority: 10 is an illustrative policy that prioritizes requests over priority 0 requests in the same pool. When using flow control, enable the EPP flowControl feature gate and have a trusted authentication layer set x-llm-d-inference-objective: interactive. Creating the Objective alone does not automatically classify all requests. Validate or replace externally supplied headers to prevent clients from self-assigning priority. This value does not reserve dedicated GPUs or define a Pod PriorityClass.

Two routing layers — keep them distinct​

An LLM inference platform separates model/provider selection from in-model Pod selection. The difference in logical roles does not by itself require two physical gateways: a supported Gateway can connect both roles using HTTPRoute together with an EPP.

Table 10. Decision question, signals, and implementations of the two routing layers (L1/L2)

LayerDecision questionSignalsUnitImplementations
L1 — LLM proxy / edge"Which provider/model/region?"cost·budget·provider health·auth·PII·semantic similaritymodel/provider targetKong, Bifrost, LiteLLM, Portkey, (kgateway)
L2 — inference routing (KV-aware)"Which GPU Pod?"vLLM internal metrics (KV cache util·prefix location·queue depth)individual PodEPP(GIE), HyperPod router, Dynamo, vLLM production-stack

GIE's official docs present combining a higher-level AI Gateway (LiteLLM, Solo AI Gateway, Apigee) on top of EPP as an integration example (not a mandate). L1 (across-model) and L2 (within-model) are complementary.

A routing "decision" is not inference — KV-aware vs context-aware

In L2 KV-cache-aware (prefix-aware) routing, the routing decision itself is not inference: it is a mechanical prefix-block hash + index lookup, with no model forward pass. Context-aware (semantic) routing, by contrast, runs an encoder/classifier model to classify intent, so lightweight inference happens on the routing path. In either case the final workload the selected Pod runs is LLM inference. See KV Cache Optimization — Cache-Aware Routing.

EPP (Endpoint Picker) — precise definition​

EPP = the GIE-defined inference-scheduler service that implements the Envoy ext-proc (External Processing) gRPC protocol. The Gateway (Envoy) calls the EPP on each request to delegate "which Pod inside the InferencePool to send to."

Three specs that make an EPP an EPP:

  1. ext-proc gRPC implementation — any gateway supporting ext-proc-style external processing can host an EPP (Istio, kgateway, Envoy Gateway, GKE Gateway, agentgateway, NGINX Gateway Fabric, etc.)
  2. Handles individual Pod addresses (podIP:port) via the InferencePool selector — Pod-level, not a Kubernetes Service
  3. Model-server metric-based scheduling — KV-cache/load aware, not a simple load balancer

EPP internal pipeline (a layered scheduler, not a single function):

Gateway vs EPP boundary: HTTPRoute pool selection, weighting, and path matching are routing performed by the Gateway (Gateway API), not part of the EPP pipeline. After the Gateway decides which InferencePool to send to, the EPP decides only "which Pod" within that pool.

Table 11. Role of each layer in the EPP internal pipeline

LayerRole
Data LayerCollects/prepares the InferencePool Pod list and vLLM metrics, and lookup data such as the prefix index
Flow ControlInferenceObjective request priority; priority·fairness·queueing (Saturation Detector overload defense)
Scheduling Layerscorer + picker — selects the actual Pod within the pool
  • scorer: the prefix-cache-scorer consumes prefix match information (PrefixCacheMatchInfo) prepared in advance by a separate producer/indexer to score each Pod. The scorer itself does not hash/index the prompt per request; block hashing and index updates are done by the data-layer producer, and the scorer reads the result to assign scores. Other scorers include KV-cache/load-aware (load·queue-depth) and LoRA affinity.
  • picker: aggregates scorer scores → final Pod selection (default max-score-picker when no picker is specified)
  • Decision delivery: x-gateway-destination-endpoint HTTP header + dynamic_metadata (both must match), one fallback can be specified

What EPP is not: not a gateway (it only makes the routing decision; the gateway proxies). Not a model router (within-model Pod selection only). The reference implementation is llm-d Router; the EPP code was consolidated into llm-d, and GIE retains only the InferencePool API and the Endpoint Picker Protocol. EPP host gateways are those supporting ext-proc-style external processing: Istio, kgateway, Envoy Gateway, GKE Gateway, agentgateway, NGINX Gateway Fabric, etc.

L2 options compared: EPP vs HyperPod Inference Operator vs Dynamo​

This compares what you implement KV-aware routing (L2) with. The common constraint is that the InferencePool selector selects only Pods in the same namespace, and the KV-aware scheduler depends on vLLM workers' real-time metrics. So the L2 scheduler is premised on living in the same cluster as vLLM; regional locality (same-region placement) is a recommendation for metric freshness and latency (a different region is not forbidden so much as discouraged because of cross-region latency).

Table 12. L2 KV-aware routing options compared (EPP, HyperPod, Dynamo)

ItemEPP(GIE) + Envoy gatewayHyperPod Inference OperatorNVIDIA Dynamo
KV-aware methodprefix-cache·kv-util scorerkvaware/prefixaware (LMCache router)KV router (Radix tree, KVPublisher/Indexer)
Managementself-managed (K8s standard)AWS-managed (EKS add-on)self-managed
Node auto-recoveryNMA+Auto Repair (separate)deep health check + spare pool, integratedseparate
Distributed inferencellm-d integrationDPD-managed (EFA/GPUDirect RDMA)prefill/decode + 3-tier KV offload
Backend enginevLLM/TGI etc.vLLM-only (kvaware is /completions only)vLLM/SGLang/TRT-LLM
CostEC2 pricingA managed premium over EC2 may apply (around +15–20% has been observed in a past ml.p5 vs p5 comparison, but it varies by instance/region/date — confirm against SageMaker pricing at adoption time)EC2 pricing
Lock-inLow (based on Gateway API/GIE standards, though an operational dependency on specific controllers/implementations remains)Medium (SageMaker CRD, but on top of vLLM/K8s standards)None
Best fitstandard·vendor-neutral·MCP integrationminimal ops·auto-recovery·governancepeak performance·multi-engine

Kong has no L2 (EPP). Kong's load balancing (consistent-hash·lowest-latency, etc.) is at the model/provider level, not Pod-level KV-aware scheduling, and being non-Envoy it does not implement GIE/ext-proc/InferencePool. If KV-aware is needed, keep Kong at L1 (edge) and use EPP/HyperPod/Dynamo for L2. (Re-check Kong release notes at adoption time — InferencePool is a gateway-neutral standard, so future support is possible.)

Kong's semantic cache ≠ vLLM KV-cache-aware routing

Kong AI Gateway's ai-semantic-cache plugin is an L1 response cache that reuses the "whole response" of semantically similar requests via embedding cosine similarity (eliminating duplicate LLM calls). This is a different layer from L2 KV-cache-aware routing, which picks a Pod by knowing the location of the vLLM Pod's prefix KV cache (three-layer cache comparison: Semantic Caching Strategy).

Also, Kong's vector store is not embedded inside Kong but connects to an external Redis/Valkey/PGVector (on AWS, IAM-authenticated ElastiCache, etc.), and embeddings themselves are generated by calling an external embedding API (OpenAI·Bedrock, etc.). AWS integration documents a Bedrock provider and the ai-aws-guardrails plugin, but direct integration with an "AWS-managed KV cache" is a separate concept. (Kong behavior differs by version, so check the Kong AI Gateway docs at adoption time.)

HyperPod's managed DPD/KV-aware is pinned to vLLM. If you need the performance ceiling of a TensorRT-LLM backend, a hybrid is possible: deploy Dynamo directly on HyperPod EKS and use HyperPod only as the node-recovery/governance layer.

Multi-region caution (e.g. Seoul ingress → Sydney inference)​

L2 (KV-aware) depends on vLLM workers' real-time metrics (freshness typically on the order of hundreds of ms to seconds), and the InferencePool selector selects only Pods in the same cluster/namespace. So the L2 scheduler is premised on being in the same cluster as vLLM. A gateway in Seoul can hardly stand in for the L2 KV-aware decision of a cluster in Sydney: as cross-region RTT grows (it varies by region pair and path — the Seoul–Sydney leg is typically measured in the tens to ~150 ms range, but measure it) metrics go stale, and above all the InferencePool cannot select Pods in another cluster. Under that premise, split roles as follows:

  • Seoul = L1 (Kong/Bifrost/kgateway): provider/region routing, auth, $budget, PII, failover, session affinity (preserving the prefix cache)
  • Sydney = L2 (EPP/HyperPod router/Dynamo) + vLLM: KV-aware Pod selection
  • The KV cache benefit occurs only inside the Sydney cluster (a cross-region hop cannot be shortened by the cache). Mind cross-region egress cost and added TTFT.

Figure 4. Multi-region L1 (Seoul) and L2 (Sydney) role separation

Expand the diagram to read labels and follow connections.
Diagram

In this layout, routing runs over AWS VPC/network-level connectivity. Seoul L1 makes the region/session decision ("this is an inference request, send to Sydney"), and on the Sydney side the gateway proxies the request while consulting the EPP (L2) over ext-proc to delegate only which Pod (the EPP returns a decision and does not proxy traffic itself — the same separation as the earlier Gateway Mode diagram). The two clusters need not be merged into a single mesh; secure a private network path via Transit Gateway inter-region peering or similar. Note, however, that Transit Gateway is a network-connectivity mechanism and does not stand in for application authentication (auth/authz is handled separately at the gateway layer), nor is it a service guaranteed to be "always one hop" (the path depends on route tables, subnets, and peering configuration, which is out of scope here). To preserve multi-turn prefix reuse, the key is session affinity: L1 consistently sends the same session to the same Sydney endpoint.


Semantic Caching​

Semantic Caching detects semantically similar prompts and reuses previous responses, which can reduce LLM API cost and latency. At the Gateway level (Bifrost/LiteLLM/Portkey) it is a response cache that determines HIT/MISS by embedding similarity; exact-match caching and KV Cache are different layers.

Threshold: a single value like 0.85 is not a universal default. The right threshold depends on the embedding model, distance metric (cosine/L2), and domain, and a high numeric score does not guarantee "same meaning" (similar wording with different meaning causes false hits). So calibrate it per model within an acceptable false-hit budget, include tenant/user/data scope in the cache key for isolation, and try an exact-match lookup first to avoid unnecessary embedding cost.

Cache layers: the Gateway's semantic response cache and vLLM's KV Cache can be combined independently. However, a provider-managed "Prompt Cache" is not a separate independent cache kind so much as a prefix cache the provider runs internally, which overlaps with the KV-cache concept (e.g. reusing a shared prefix). The three are therefore not fully orthogonal — distinguish what each cache keys on and which layer it operates at.

Design principles (3-tier cache comparison, similarity threshold tradeoffs, tool comparison table, cache key design, observability·production checklist) are covered in detail in separate documentation.


agentgateway Data Plane​

Overview​

agentgateway is an AI workload-dedicated data plane for kgateway. Traditional Envoy is optimized for stateless HTTP/gRPC, but AI agents have special requirements like stateful JSON-RPC sessions, MCP protocols, and Tool Poisoning prevention.

Envoy vs agentgateway Comparison​

Table 13. Envoy data plane vs agentgateway

ItemEnvoy Data Planeagentgateway
Session ManagementStateless, HTTP cookie-basedMCP session tracking (Mcp-Session-Id), optional session store — sessions are optional per the MCP spec and stateless operation is allowed
ProtocolsHTTP/1.1, HTTP/2, gRPCMCP (Model Context Protocol), A2A (Agent-to-Agent)
SecuritymTLS, RBACTool Poisoning prevention, per-session Authorization
RoutingPath/header-basedSession ID-based, tool call validation
ObservabilityHTTP metrics, Access LogLLM token tracking, tool call chains, cost

Core Features​

1. Session Management: MCP Streamable HTTP (2025-06-18 spec) tracks sessions via the Mcp-Session-Id header (not X-MCP-Session-ID). Session management is optional per the spec, and servers may operate statelessly. Where agentgateway does sticky-session routing or inactive-session cleanup, the timeout value (e.g. 30 minutes) is a configurable value, not a spec-fixed default; confirm the applied value in your deployment configuration.

2. MCP/A2A Protocol Support: routes MCP and A2A traffic. Paths such as /mcp/v1 and /a2a/v1 are example paths you configure in deployment, not spec-mandated fixed paths (the actual endpoint path is set in configuration).

3. Tool guardrails (require configuration): an allowed-tool list (allowlist), blocking of dangerous tools, response size limits, and integrity verification can be set as policy. This is not a guarantee that Tool Poisoning is "automatically and fully prevented" or that SHA-256 verification "always" runs — the applied scope depends on which guardrail policies the operator configures. See the agentgateway MCP guardrails post.

4. Authorization: per-session authorization policies such as JWT token verification and role-based tool access can be configured. Session-hijacking prevention is an effect of configuring these authentication/authorization policies correctly, not something guaranteed automatically without configuration.

agentgateway Project Status

agentgateway is an AI-dedicated data plane separated from the kgateway project in late 2025, currently under active development. Features are continuously added to keep pace with rapid evolution of MCP and A2A protocols.


Monitoring & Observability​

Core Metrics​

The table below uses the exact Envoy admin-statistic names, including their configurable tree prefixes, from the Envoy v1.36.0 HTTP connection-manager reference and cluster reference, checked on 30 October 2026. <stat_prefix>, <name>, and <code> are placeholders for the configured HTTP prefix, upstream cluster, and response code. Dotted admin names are not PromQL series names.

The kgateway metrics documentation snapshot shows envoy_cluster_external_upstream_rq{envoy_response_code="200",envoy_cluster_name="kube_httpbin_httpbin_8000"} as a Prometheus example. Its listener queries use envoy_http_downstream_rq_total with envoy_http_conn_manager_prefix; prefix values depend on the kgateway version and listener configuration. Inspect the deployed version, tag extraction, histogram units, and actual /stats/prometheus scrape before creating queries or recording rules. The default ?usedonly filtering can omit zero-valued series; an absent series is not automatically zero traffic. These are documented contracts, not a scrape from this report's deployment.

📊 Envoy v1.36.0 admin statistic names
MetricDescriptionUsage
http.<stat_prefix>.downstream_rq_total
Total request countTraffic monitoring
http.<stat_prefix>.downstream_rq_time
Total request/response time histogram (milliseconds)Latency analysis
cluster.<name>.upstream_rq_<code>
Backend response codesError tracking
cluster.<name>.upstream_cx_active
Active connectionsCapacity planning
cluster.<name>.upstream_rq_retry
Retry countStability analysis

Table 14. Core inference-gateway metric categories and their meaning

Metric CategoryKey ItemMeaning
LatencyTTFT (Time to First Token)Time until first token generation. User-perceived responsiveness
ThroughputTPS (Tokens Per Second)Tokens generated per second. Model serving efficiency
Error Rate5xx / Total RequestsBackend failure ratio (the threshold below is a starter example)
Cache Hit RateCache Hit / Total RequestsSemantic Cache efficiency (target depends on workload)
CostΣ (usage × unit price) for input/output/cached tokens separatelyInput/output/cached token prices differ, so account for them separately

The thresholds above are starting-point examples, not universal defaults. Error-rate, cache-hit, and P99 targets vary with traffic volume, measurement window, minimum sample size, and endpoint scope, so define the metric type/labels/window and tune to each service's SLO. Cost must be computed per token type because input/output/cached token prices differ and effective cost depends on the cache hit rate.

Langfuse Integration​

Send traces from Bifrost/LiteLLM to Langfuse to track prompts/completions, token usage, cost, and tool call chains. In LiteLLM there are two distinct paths:

  • OpenTelemetry path (recommended): the langfuse_otel preset (OpenTelemetry v2), recommended for Langfuse v3/v4; it sends traces to the OTLP endpoint.
  • SDK-callback path (legacy): success_callback: ["langfuse"] is a callback that uses the Langfuse Python SDK (langfuse>=4.7,<5). It is not the same as "sending OTel traces," and requires configuring the credentials (LANGFUSE_PUBLIC_KEY·LANGFUSE_SECRET_KEY·LANGFUSE_HOST) and SDK version.

Bifrost enables OTel export via its otel plugin. For detailed configuration, see the LiteLLM Langfuse integration docs and Monitoring Stack Setup.

The conditions below are starting-point examples, not universal defaults. Define a measurement window, minimum sample size, and target endpoint scope for each, express them in PromQL against the actual metric names/labels (e.g. the series Envoy/kgateway exports), and tune to your service SLO.

Table 15. Example alert rules and severity

AlertCondition (example)Severity
High error rate5xx ratio > 5% (5-min window, once minimum request count is met)Critical
High latencyP99 > 30s (5-min window)Warning
Circuit breaker / outlier activerelevant state metric == 1Critical
Cache hit rate dropCache hit < target (workload-specific)Warning
Budget approachingBudget > 80%Warning

Production Deployment Guides​

For actual code examples and YAML manifests, refer to Reference Architecture section:

Cost and Observability​

Governance & Tenancy​


References​

Official Documentation​

LLM Providers​

Research Papers & Patterns​