Skip to main content

Inference Gateway Deployment Guide

15 min read
AI documentation guide

This document covers production deployment procedures for kgateway + Bifrost-based inference gateway. For architecture concepts and routing strategies (Cascade, Semantic Router, 2-Tier structure), refer to Inference Gateway Routing.

Guide Structure

This guide consists of 3 documents. Learn sequentially or reference specific sections as needed. The learning, deployment and total times listed below are author's estimates, not measured results, and assume that everything in "Prerequisites" is already in place. Elapsed time does not mean a deployment is complete; judge completion by the checks in each document (resource status and a successful inference response).

Production Inference Pipeline Reference Architecture​

The complete request flow of an EKS-based production inference pipeline. CloudFront (WAF/Shield) → NLB → kgateway receives inbound traffic, after which you compose one of two routing paths by purpose.

  • L1 — across-model proxy / governance: Bifrost (or the LLM Classifier) abstracts external/internal models and provides cascade, cost tracking, and semantic cache. It decides "which model." → Basic Deployment · Advanced Features
  • L2 — within-model KV-aware routing: the Gateway API Inference Extension (InferencePool + EPP) selects a Pod based on vLLM metrics (KV cache and load). It decides "which Pod." → Advanced Features: Inference Extension

The two layers are not exclusive and can be used together (L1 → L2 → vLLM). For the layer definitions and selection criteria, refer to Routing Strategy.

About the reference architecture diagram

The diagram below is a target (reference) architecture that includes L1 (Bifrost), L2 (llm-d EPP), and observability all together. An actual deployment selects and composes the two paths above; when KV-aware (L2) is needed, follow the InferencePool + EPP procedure in the Inference Extension section. The components and the top badge in the diagram show the target configuration; no record of deploying and verifying this whole configuration end to end (E2E) is attached to this document.

Deployment Stages Overview​

Basic Deployment (Required)​

Configure kgateway + HTTPRoute + Bifrost behind a single NLB endpoint to complete basic inference pipeline.

Includes:

  • kgateway installation and Gateway API CRD configuration
  • GatewayClass, Gateway, HTTPRoute resource definitions
  • Cross-namespace access via ReferenceGrant
  • Bifrost Gateway Mode configuration (config.json + PVC)
  • provider/model format and IDE compatibility (Aider, Cline, Continue.dev)
  • config.json change procedure (restart-based reconciliation, preserves stored config)

Learning Time: 30 min | Deployment Time: 45 min

Advanced Features (Optional)​

Add prompt-based automatic routing, production security layer, and Semantic Caching to enhance cost optimization and security.

Includes:

  • LLM Classifier deployment (prompt-based SLM/LLM automatic branching)
  • Gateway API Inference Extension (InferencePool + EPP) — KV-aware (L2) routing
  • CloudFront + WAF/Shield security layer
  • Semantic Caching implementation options (GPTCache, RedisVL, Portkey, Helicone)

Learning Time: 45 min | Deployment Time: 60-90 min

Troubleshooting (Reference)​

Common issues and solutions during deployment and operations.

Includes:

  • 404 Not Found (HTTPRoute/Gateway configuration errors)
  • Bifrost provider/model errors
  • Bifrost model name normalization issues
  • Langfuse Sub-path 404
  • OTel Trace not arriving

Reference Frequency: During deployment or when issues occur

Learning Paths​

Quick Start (Development/Test Environment)​

Figure 1. Quick-start learning path (Basic Deployment → Troubleshooting)

Expand the diagram to read labels and follow connections.
Diagram
  1. Configure kgateway + Bifrost with Basic Deployment
  2. Refer to Troubleshooting when issues occur

Time Required: 1-2 hours

Production Configuration (Complete Pipeline)​

Figure 2. Production configuration learning path (Basic Deployment → Advanced Features → Troubleshooting)

Expand the diagram to read labels and follow connections.
Diagram
  1. Configure basic infrastructure with Basic Deployment
  2. Add LLM Classifier + CloudFront/WAF + Semantic Caching in Advanced Features
  3. Refer to Troubleshooting during operations

Time Required: 3-4 hours

Prerequisites​

Verify the following before proceeding with all deployment stages.

Required​

  • EKS cluster (K8s 1.33+ recommended; core DRA (Dynamic Resource Allocation) reached GA in K8s 1.34)
  • kubectl installed with cluster access
  • Helm 3.x installed
  • vLLM or llm-d based model serving Pods deployed
  • AWS Load Balancer Controller installed (for automatic NLB creation)
  • Langfuse deployed (refer to Langfuse Deployment Guide)
  • Production environment: ACM certificate issued (for CloudFront + TLS)

Next Steps​

References​