Skip to main content

Inference Gateway & LLM Gateway Routing Strategy

38 min read
AI documentation guide

This document covers design principles for 2-Tier gateway architecture and routing strategies (Cascade / Semantic Router / Hybrid). For actual deployment procedures including Helm installation, HTTPRoute manifests, and OTel integration, refer to Inference Gateway Deployment Guide.

Overview​

In large-scale AI model serving environments, distinguishing infrastructure traffic management from LLM provider abstraction helps clarify responsibilities. The 2-Tier layout below is a design option for separating ownership, policies, and scaling; it is not a requirement for two traffic gateways in every llm-d deployment.

2-Tier Gateway Architecture:

  • L1 (Ingress Gateway): kgateway — Kubernetes Gateway API standard, traffic routing, mTLS, rate limiting
  • L2-A (Inference Gateway): Bifrost/LiteLLM — Provider integration, cascade routing, semantic caching
  • L2-B (Data Plane): agentgateway — MCP/A2A protocols, stateful session management

Each tier is managed independently, separating infrastructure and AI workloads.

2-Tier Gateway Architecture​

Gateway layer definitions are unified in a dedicated document

The platform-wide gateway-layer terminology and role definitions are consolidated in Tiered Gateway Architecture. This document focuses on the routing strategy of Tier 2-A (LLM API Gateway). For in-cluster inference pod routing (Tier 2 ① Inference Extension), see the Gateway API Inference Extension section below.

Gateway Layer Separation​

LLM inference platforms must clearly distinguish 3 different Gateway roles. (For the full layer definitions, see Tiered Gateway Architecture)

Gateway TypeRoleImplementationLocation
Ingress GatewayExternal traffic ingress, TLS termination, path-based routingkgateway (NLB integration)Tier 1
LLM API GatewayModel selection, intelligent routing, request cascading (external/internal model abstraction)Bifrost / LiteLLMTier 2-A
Agent Data PlaneMCP/A2A protocols, stateful sessions, tool routingagentgatewayTier 2-B

Terminology note: Here, Tier 2-A "LLM API Gateway" is a provider proxy (Bifrost/LiteLLM) that abstracts the model API. It serves a different purpose from the Gateway API Inference Extension (Tier 2 ①, later in this document) that routes to in-cluster inference pods.

Expand the diagram to read labels and follow connections.
Diagram

Core Principles:

  • Ingress Gateway (kgateway): Handles network-level traffic control only. Does not include model selection logic
  • LLM API Gateway (Bifrost/LiteLLM): Analyzes request complexity → Automatically selects appropriate model → Cost optimization
  • Agent Data Plane (agentgateway): Handles AI-specific protocols (MCP/A2A), maintains stateful sessions

Overall Architecture​

Expand the diagram to read labels and follow connections.
Diagram

Responsibility Separation by Tier​

TierComponentResponsibilityProtocol
Tier 1 (Ingress Gateway)kgateway (Envoy-based)Traffic routing, mTLS, rate limiting, network policiesHTTP/HTTPS, gRPC
Tier 2-A (LLM API Gateway)Bifrost / LiteLLMIntelligent model selection, cost tracking, request cascading, semantic cachingOpenAI-compatible API
Tier 2-B (Agent Data Plane)agentgatewayMCP/A2A session management, self-hosted inference routing, Tool Poisoning preventionHTTP, JSON-RPC, MCP, A2A

Traffic Flow​

External LLM: Client → kgateway → Bifrost/LiteLLM (Cascade + Cache) → OpenAI → Response + Cost tracking Self-hosted vLLM: Client → kgateway → agentgateway → vLLM → Response

kgateway (L1 Inference Gateway)​

Gateway API-based Routing​

kgateway implements the Kubernetes Gateway API standard, enabling vendor-neutral configuration.

ComponentRoleDescription
GatewayClassGateway implementation definitionDesignate Kgateway controller
GatewayEntry point definitionConfigure listeners, TLS, addresses
HTTPRouteRouting rulesPath, header-based routing
BackendModel servicevLLM, TGI and other inference servers

Gateway API v1.2.0+ provides HTTPRoute improvements, GRPCRoute stabilization, and BackendTLSPolicy, fully supported by kgateway v2.0+.

Dynamic Routing Concepts​

Routing TypeCriteriaUse Case
Header-basedx-model-id, x-providerBackend selection by model/provider
Path-based/v1/chat/completions, /v1/embeddingsService separation by API type
Weight-basedbackendRef weightCanary deployment, A/B testing
Composite conditionsHeaders + Path + TierPremium/standard customer backends

Canary deployments start with 5-10% traffic and gradually increase, with immediate rollback via weight=0 on issues.

Load Balancing Strategies​

StrategyDescriptionSuitable Scenario
Round RobinSequential distribution (default)Uniform model instances
RandomRandom distributionLarge backend pools
Consistent HashSame key → Same backendKV Cache reuse, session affinity

Consistent Hash is particularly useful for LLM inference. Routing requests from the same user to the same vLLM instance increases prefix cache hit rates, significantly improving TTFT (Time to First Token).

Topology-Aware Routing (Kubernetes 1.33+)​

Kubernetes 1.33+ topology-aware routing prioritizes same-AZ Pod communication to reduce cross-AZ data transfer costs.

🚀 Topology-Aware Routing Effects
MetricBeforeTopology-AwareImprovement
Cross-AZ TrafficHighMinimized
50% data transfer cost savings
LatencyHigh (cross-AZ)Low (same AZ)
30-40% P99 latency improvement
Network BandwidthLimitedOptimized
20-30% throughput increase

Failure Handling Concepts​

MechanismDescriptionLLM Inference Considerations
TimeoutMaximum processing time per requestLLM long response generation takes tens of seconds. Adequate timeout needed (120s+)
RetryAuto-retry on 5xx, timeout, connection failureMax 3 retries. Infinite retries cause system overload
Circuit BreakerTemporarily block backend on consecutive failuresSet maxEjectionPercent to 50% or below to ensure at least half backends available

For streaming responses, backendRequest timeout is for first byte, request is for total time. POST retries require idempotency guarantees (caution with tool calls).

LLM Gateway Solution Comparison​

Major Solution Comparison Table​

SolutionLanguageKey FeaturesCascade RoutingLicenseBest For
BifrostGo50x faster, CEL Rules conditional routing, failoverCEL Rules + external classifierApache 2.0High performance, low cost, self-hosted
LiteLLMPython100+ providers, native complexity-based routingrouting_strategy: complexity-basedMITPython ecosystem, rapid prototyping
vLLM Semantic RouterPythonvLLM-only, lightweight embedding-based routingEmbedding similarity-basedApache 2.0vLLM standalone environment
PortkeyTypeScriptSOC2 certified, semantic caching, Virtual KeysSupportedProprietary + OSSEnterprise, compliance
Kong AI GatewayLua/CMCP support, leverages existing Kong infraPluginApache 2.0 / EnterpriseExisting Kong users
HeliconeRustGateway + Observability integrated, high performanceSupportedApache 2.0High performance + observability needed
OpenRouterSaaS (hosted)Unified API for 400+ models·60+ providers, provider fallback, OpenAI-compatibleProvider routing supportedSaaS (commercial)Fast multi-provider integration, prototyping
LiteLLM and Kong Are Either/Or

LiteLLM and Kong AI Gateway are both L1 gateways — choose one of the two. There is no verified reference for an architecture combining both products (e.g., Kong in front + LiteLLM behind). For selection criteria from the tenancy-model and budget-enforcement perspective, see AI Gateway Multi-Tenancy — Selection Criteria.

Connecting L1 Budget Signals to Fallback Routing

Among L1 signals, budget is directly tied to governance policy. When a tenant's budget is exhausted, the L1 gateway can block the request (hard budget) or take a fallback path that downgrades to a lower-cost model. For per-tenant budget hierarchies and enforcement, see AI Gateway Multi-Tenancy; for the block/alert/fallback policy matrix on budget overrun, see LLM FinOps Chargeback. These policy decisions all happen at L1; L2 (KV-aware Pod selection) does not handle budget signals.

Self-hosted vs SaaS

In the table above, Bifrost·LiteLLM·Helicone·vLLM Semantic Router are self-hosted (deployed in-cluster), whereas OpenRouter is hosted SaaS. SaaS provides instant access to 400+ models and delegates provider fallback/billing, which is advantageous for fast integration and prototyping. However, since prompts are sent to an external service, review the governance considerations for environments with data sovereignty or regulatory requirements.

Bifrost vs LiteLLM​

Bifrost: Go implementation with 50x faster throughput than Python, 1/10 memory usage. CEL Rules enable conditional routing (header-based cascade, failover). Helm Chart deployment, OpenAI-compatible API. Proxy latency < 100us. Intelligent cascade via app-side complexity score calculation → x-complexity-score header → CEL rule branching pattern or Go Plugin.

LiteLLM: 100+ provider support, native complexity-based routing (activate with 1-line routing_strategy: complexity-based config), one-line Langfuse integration (success_callback: ["langfuse"]), direct LangChain/LlamaIndex integration. However, Python-based with lower throughput, higher memory usage.

Selection Criteria​

Use CaseRecommended SolutionReason
Intelligent cascade (convenience priority)LiteLLMNative complexity-based routing, 1-line config
Intelligent cascade (performance priority)BifrostCEL Rules + external classifier, 50x faster
vLLM standalone environmentvLLM Semantic RoutervLLM native, lightweight routing
High performance, low cost self-hostedBifrost50x faster processing, low memory
Python ecosystem (LangChain)LiteLLMNative integration, 100+ providers
Enterprise compliancePortkeySOC2/HIPAA/GDPR, Semantic Cache
High performance + observability integratedHeliconeRust-based All-in-one
ScenarioRecommended StackReason
Startup/PoCkgateway + LiteLLMLow cost, 10-min deployment, complexity routing 1-line
Self-hosted focused (performance)kgateway + Bifrost (CEL cascade) + agentgatewayHigh performance, external+self-hosted pool 2-Tier
Enterprise multi-providerkgateway + Portkey + LangfuseCompliance, 250+ providers
Hybrid (external+self-hosted)kgateway + Bifrost/LiteLLM + agentgatewayExternal via Bifrost/LiteLLM, self-hosted via agentgateway
Global deploymentCloudflare AI Gateway + kgatewayEdge caching, DDoS protection

Request Cascading: Intelligent Model Routing​

Request Cascading is an intelligent optimization technique that automatically analyzes request complexity and routes to the appropriate model. The three patterns (weight-based, fallback-based, intelligent routing), implementation approach comparison (LLM Classifier, LiteLLM, vLLM Semantic Router), RouteLLM research reference, and cost savings are covered in detail in Request Cascading — Intelligent Model Routing.

For threshold/keyword tuning and misroute detection operations, see Cascade Routing Tuning.

Gateway API Inference Extension​

Kubernetes Gateway API enables managing LLM inference as Kubernetes-native resources through Inference Extension.

Core CRDs (Custom Resource Definitions)​

The reviewed baseline is llm-d v0.8.1 (2026-06-26), router v0.9.0, GIE v1.5.0, and Gateway API v1.5.1. Primary schemas: llm-d v0.8.1, InferencePool v1, InferenceObjective v1alpha2, HTTPRoute v1.

ResourceOwner / APIResponsibilityExample
Deployment / LeaderWorkerSetKubernetes / separate workload APIImages, model arguments, Pod GPU requests, replica countsWorkload replicas: 3 and nvidia.com/gpu
InferencePoolGIE, inference.networking.k8s.io/v1Select same-namespace Pods and define ports/EPP referenceselector.matchLabels, targetPorts, endpointPickerRef
InferenceObjectivellm-d, llm-d.ai/v1alpha2 (alpha, optional)Request priority for a specific poolpoolRef, integer priority: 10
HTTPRouteGateway API, gateway.networking.k8s.io/v1Select pools by path, header, or weightbackendRefs.kind: InferencePool

InferenceModel is the historical name of the earlier policy API. This baseline uses llm-d InferenceObjective. criticality: high, model images, GPU requests, and replica counts are not InferenceObjective/InferencePool fields. Distinguish the GIE v1 InferencePool API from the llm-d alpha policy API.

There is no LLMRoute kind in the reviewed GIE or llm-d Router CRDs. Even the router's llmroute.yaml test file uses kind: HTTPRoute. Do not treat a filename as an API kind or copy that test file's old API group.

Gateway API Inference Extension Integration​

In Gateway Mode, one compatible Gateway can handle both traditional Services and InferencePools. The EPP returns its selection to the Gateway and does not proxy inference traffic itself. A separate edge Gateway is an option when retaining existing ingress or separating security/ownership; account for extra hops, operational cost, and consistent timeout, retry, streaming, and authentication-header handling.

Solid lines show requests/responses and EPP calls; dotted lines show configuration references.

Expand the diagram to read labels and follow connections.
Diagram

This example defines routing resources only. It requires an existing llm-d namespace, inference-gateway Gateway, inference-epp Service (EPP on port 9002 configured for gpu-pool), and model-server Pods labeled app: vllm listening on port 8000. Install Gateway API v1.5.1, the GIE v1.5.0 InferencePool CRD, the router v0.9.0 InferenceObjective CRD, and compatible controllers.

apiVersion: inference.networking.k8s.io/v1
kind: InferencePool
metadata:
  name: gpu-pool
  namespace: llm-d
spec:
  selector:
    matchLabels:
      app: vllm
  targetPorts:
    - number: 8000
  endpointPickerRef:
    name: inference-epp
    kind: Service
    port:
      number: 9002
    failureMode: FailClose
---
apiVersion: llm-d.ai/v1alpha2
kind: InferenceObjective
metadata:
  name: interactive
  namespace: llm-d
spec:
  poolRef:
    group: inference.networking.k8s.io
    kind: InferencePool
    name: gpu-pool
  priority: 10
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: inference-route
  namespace: llm-d
spec:
  parentRefs:
    - name: inference-gateway
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /v1
      backendRefs:
        - group: inference.networking.k8s.io
          kind: InferencePool
          name: gpu-pool
          port: 8000

priority: 10 is an illustrative policy that prioritizes requests over priority 0 requests in the same pool. When using flow control, enable the EPP flowControl feature gate and have a trusted authentication layer set x-llm-d-inference-objective: interactive. Creating the Objective alone does not automatically classify all requests. Validate or replace externally supplied headers to prevent clients from self-assigning priority. This value does not reserve dedicated GPUs or define a Pod PriorityClass.

Semantic Caching​

Semantic Caching detects semantically similar prompts and reuses previous responses, simultaneously reducing LLM API costs and latency. At the Gateway level (Bifrost/LiteLLM/Portkey), HIT/MISS is determined by embedding similarity, so it can be combined independently with KV Cache (vLLM) · Prompt Cache (provider-managed).

Recommended default threshold: 0.85 — allows same meaning, different expression

Design principles (3-tier cache comparison, similarity threshold tradeoffs, tool comparison table, cache key design, observability·production checklist) are covered in detail in separate documentation.

agentgateway Data Plane​

Overview​

agentgateway is an AI workload-dedicated data plane for kgateway. Traditional Envoy is optimized for stateless HTTP/gRPC, but AI agents have special requirements like stateful JSON-RPC sessions, MCP protocols, and Tool Poisoning prevention.

Envoy vs agentgateway Comparison​

ItemEnvoy Data Planeagentgateway
Session ManagementStateless, HTTP cookie-basedStateful JSON-RPC sessions, in-memory session store
ProtocolsHTTP/1.1, HTTP/2, gRPCMCP (Model Context Protocol), A2A (Agent-to-Agent)
SecuritymTLS, RBACTool Poisoning prevention, per-session Authorization
RoutingPath/header-basedSession ID-based, tool call validation
ObservabilityHTTP metrics, Access LogLLM token tracking, tool call chains, cost

Core Features​

1. Stateful JSON-RPC Session Management: X-MCP-Session-ID header-based session tracking, Sticky Session routing, automatic inactive session cleanup (default 30 minutes)

2. Native MCP/A2A Protocol Support: /mcp/v1 (MCP protocol), /a2a/v1 (A2A agent communication) path support

3. Tool Poisoning Prevention: Allowed tool list, dangerous tool blocking (exec_shell, read_credentials), response size limits, integrity verification (SHA-256)

4. Per-session Authorization: JWT token verification, role-based tool access, session hijacking prevention

agentgateway Project Status

agentgateway is an AI-dedicated data plane separated from the kgateway project in late 2025, currently under active development. Features are continuously added to keep pace with rapid evolution of MCP and A2A protocols.

Monitoring & Observability​

Core Metrics​

Core metrics to monitor in AI inference gateways:

📊 Kgateway Prometheus Metrics
MetricDescriptionUsage
kgateway_requests_total
Total request countTraffic monitoring
kgateway_request_duration_seconds
Request processing timeLatency analysis
kgateway_upstream_rq_xx
Backend response codesError tracking
kgateway_upstream_cx_active
Active connectionsCapacity planning
kgateway_retry_count
Retry countStability analysis
Metric CategoryKey ItemMeaning
LatencyTTFT (Time to First Token)Time until first token generation. User-perceived responsiveness
ThroughputTPS (Tokens Per Second)Tokens generated per second. Model serving efficiency
Error Rate5xx / Total RequestsBackend failure ratio. Immediate action if > 5%
Cache Hit RateCache Hit / Total RequestsSemantic Cache efficiency. 30%+ recommended
CostToken usage by model × unit priceReal-time cost tracking

Langfuse OTel Integration​

Send OTel traces from Bifrost/LiteLLM to Langfuse to track prompts/completions, token usage, cost analysis, and tool call chains. Bifrost activates via otel plugin, LiteLLM via success_callback: ["langfuse"] config. For detailed configuration, refer to Monitoring Stack Setup.

AlertConditionSeverity
High error rate5xx > 5% (5 min)Critical
High latencyP99 > 30s (5 min)Warning
Circuit breaker activatedcircuit_breaker_open == 1Critical
Cache hit rate dropCache hit < 30%Warning
Budget approachingBudget > 80%Warning

Production Deployment Guides​

For actual code examples and YAML manifests, refer to Reference Architecture section:

Cost and Observability​

Governance & Tenancy​

References​

Official Documentation​

LLM Providers​

Research Papers & Patterns​