Skip to main content

Migration Execution Strategy

12 min read
AI documentation guide
info

This document extends the Gateway API Adoption Guide. It provides a practical strategy for migrating from NGINX Ingress to Gateway API.

1. Prerequisites: CRD Installation​

The Gateway API CRD version must match the selected controller’s supported version. The common example below uses v1.4.1 for the linked Cilium 1.19.3 guide. Do not overwrite Standard/Experimental bundles with different releases. Other controllers and the inference Gateway in section 9.4 of the Cilium guide have their own compatibility requirements.

1.1 Standard Gateway API CRDs​

# Standard CRDs for the Cilium 1.19.3 baseline
kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.1/standard-install.yaml

# Select the same-version experimental bundle instead if experimental resources are required
# kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/v1.4.1/experimental-install.yaml

Installed CRDs:

  • gatewayclasses.gateway.networking.k8s.io
  • gateways.gateway.networking.k8s.io
  • httproutes.gateway.networking.k8s.io
  • referencegrants.gateway.networking.k8s.io
  • grpcroutes.gateway.networking.k8s.io (Standard)
  • tcproutes.gateway.networking.k8s.io (Experimental)
  • tlsroutes.gateway.networking.k8s.io (Experimental)
  • udproutes.gateway.networking.k8s.io (Experimental)

1.2 Additional Installation for Each Controller​

These are controller-specific installation templates, not a claim that all combinations were verified with v1.4.1. The official AWS LBC v3.0.0 guide targets Gateway API v1.3.0 and requires additional LBC Gateway CRDs. Select CRD and chart versions together using the chosen NGINX/Envoy release’s installation and compatibility documentation as well. Do not run all controller installation alternatives sequentially. AWS Native (ALB + NLB Gateway)

# Install AWS Load Balancer Controller v3.0+ with Gateway API support
helm repo add eks https://aws.github.io/eks-charts
helm repo update

# Create IRSA (IAM Role for Service Account)
eksctl create iamserviceaccount \
  --cluster=<cluster-name> \
  --namespace=kube-system \
  --name=aws-load-balancer-controller \
  --role-name AmazonEKSLoadBalancerControllerRole \
  --attach-policy-arn=arn:aws:iam::<account-id>:policy/AWSLoadBalancerControllerIAMPolicy \
  --approve

# First create the official IAM policy in your account and select a compatible chart version
# https://kubernetes-sigs.github.io/aws-load-balancer-controller/latest/deploy/installation/
: "${LBC_CHART_VERSION:?Set a chart version compatible with the chosen Gateway API CRDs}"
# Install with Helm
helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
  --version "$LBC_CHART_VERSION" -n kube-system \
  --set clusterName=<cluster-name> \
  --set serviceAccount.create=false \
  --set serviceAccount.name=aws-load-balancer-controller \
  --set controllerConfig.featureGates.ALBGatewayAPI=true \
  --set controllerConfig.featureGates.NLBGatewayAPI=true

# Verify the installation
kubectl get deployment -n kube-system aws-load-balancer-controller

NGINX Gateway Fabric

# Install NGINX Gateway Fabric
kubectl apply -f https://github.com/nginxinc/nginx-gateway-fabric/releases/download/v1.6.0/crds.yaml
kubectl apply -f https://github.com/nginxinc/nginx-gateway-fabric/releases/download/v1.6.0/nginx-gateway.yaml

# Verify the installation
kubectl get pods -n nginx-gateway
kubectl get gatewayclass nginx

Envoy Gateway

# Install Envoy Gateway
helm install eg oci://docker.io/envoyproxy/gateway-helm \
  --version v1.3.0 \
  --namespace envoy-gateway-system \
  --create-namespace

# Verify the installation
kubectl get pods -n envoy-gateway-system
kubectl get gatewayclass envoy-gateway

Cilium Gateway API

No separate installation is required when gatewayAPI.enabled=true was already set during Cilium installation.

# Check the GatewayClass
kubectl get gatewayclass cilium

2. 5-Phase Migration Process​

Expand the diagram to read labels and follow connections.
Diagram

3. Phase-by-Phase Detail​

Step 1.1: Collect the current Ingress inventory

# Export all Ingress resources
kubectl get ingress -A -o json > ingress-inventory.json

# Summarize key information
cat ingress-inventory.json | jq -r '
  .items[] |
  {
    namespace: .metadata.namespace,
    name: .metadata.name,
    class: .spec.ingressClassName,
    hosts: [.spec.rules[].host],
    paths: [.spec.rules[].http.paths[].path],
    tls: (.spec.tls != null)
  }
' > ingress-summary.json

# Summarize statistics
echo "=== Ingress Statistics ==="
echo "Total Ingress: $(cat ingress-inventory.json | jq '.items | length')"
echo "With TLS: $(cat ingress-inventory.json | jq '[.items[] | select(.spec.tls != null)] | length')"
echo "Unique Hosts: $(cat ingress-inventory.json | jq -r '[.items[].spec.rules[].host] | unique | length')"

Step 1.2: Map features from NGINX Ingress to Gateway API

🔄 NGINX Ingress → Gateway API Migration Mapping
Gateway API alternatives for existing Ingress features
host: example.com→HTTPRoute.spec.hostnames
DirectDirect mapping
path: /api→HTTPRoute.spec.rules[].matches[].path
DirectDirect mapping
pathType: Prefix→path.type: PathPrefix
IdenticalIdentical
annotations: rewrite-target→filters[].type: URLRewrite
StandardizedStandardized
annotations: rate-limit→Policy Attachment (varies by implementation)
In ProgressStandardization in progress
annotations: cors-*→Policy Attachment
In ProgressStandardization in progress
annotations: auth-*→Policy Attachment or external auth
ExternalOAuth2-Proxy etc. recommended
annotations: ssl-redirect→Gateway TLS listener auto-handling
AutomatedAutomated

Step 1.3: Assess risks

# risk-assessment.yaml
risks:
  - id: RISK-001
    category: Missing functionality
    description: "No direct alternative to the NGINX rate-limit annotation"
    severity: MEDIUM
    mitigation: "Use AWS WAF or the Envoy Rate Limit service"

  - id: RISK-002
    category: Downtime
    description: "Migration to Cilium ENI mode causes downtime"
    severity: HIGH
    mitigation: "Use a blue-green cluster cutover or schedule a maintenance window"

  - id: RISK-003
    category: Learning curve
    description: "The team has limited Gateway API experience"
    severity: LOW
    mitigation: "Allow sufficient testing time during the Phase 2 PoC"

4. Validation Script​

This is a focused check for HTTP/HTTPS routes with selector-backed Service backends. It requires Bash, jq, kubectl, and curl. Supply a host and path for the selected listener and Route; test additional header, method, or authentication requirements separately. A successful response does not verify every traffic case or backend identity.

#!/bin/bash
# validate-httproute.sh: selector-backed Service routes, explicit Gateway/listener and probe.
set -euo pipefail
if [ "$#" -ne 7 ]; then
  echo "Usage: $0 <route-namespace> <route-name> <gateway-namespace> <gateway-name> <listener> <probe-host> <probe-path>" >&2
  exit 1
fi
NS=$1; ROUTE_NAME=$2; GW_NS=$3; GW_NAME=$4; LISTENER=$5; HOST=$6; PROBE_PATH=$7
[[ "$HOST" != *'*'* && "$PROBE_PATH" == /* ]] || { echo "Use a concrete hostname and absolute path" >&2; exit 1; }
ROUTE=$(kubectl get httproute "$ROUTE_NAME" -n "$NS" -o json)
GW=$(kubectl get gateway "$GW_NAME" -n "$GW_NS" -o json)
CLASS=$(jq -r '.spec.gatewayClassName' <<<"$GW")
CONTROLLER=$(kubectl get gatewayclass "$CLASS" -o jsonpath='{.spec.controllerName}')

# Gateway readiness is distinct from the HTTPRoute parent status; reject stale conditions.
jq -e '.metadata.generation as $g | .status.conditions as $c |
  all(["Accepted", "Programmed"][]; . as $t |
    any($c[]?; .type == $t and .status == "True" and .observedGeneration == $g))' <<<"$GW" >/dev/null
LISTENER_JSON=$(jq -ce --arg l "$LISTENER" '.spec.listeners[] | select(.name == $l)' <<<"$GW")
PORT=$(jq -r '.port' <<<"$LISTENER_JSON")
PROTOCOL=$(jq -r '.protocol' <<<"$LISTENER_JSON")
case "$PROTOCOL" in HTTP) SCHEME=http ;; HTTPS) SCHEME=https ;; *) echo "Only HTTP/HTTPS probes are supported" >&2; exit 1 ;; esac

# Match the intended parent and its controller, including namespace, sectionName and port.
jq -e --arg ns "$NS" --arg gn "$GW_NAME" --arg gns "$GW_NS" --arg l "$LISTENER" \
  --arg c "$CONTROLLER" --argjson port "$PORT" '
  def canonical: {group:(.group // "gateway.networking.k8s.io"), kind:(.kind // "Gateway"),
    namespace:(.namespace // $ns), name, sectionName:(.sectionName // null), port:(.port // null)};
  .metadata.generation as $g |
  [.spec.parentRefs[] | canonical | select(.group == "gateway.networking.k8s.io" and .kind == "Gateway"
    and .name == $gn and .namespace == $gns and (.sectionName == null or .sectionName == $l)
    and (.port == null or .port == $port))] as $refs |
  any(.status.parents[]?; . as $p | (.parentRef | canonical) as $ref |
    .controllerName == $c and any($refs[]; . == $ref) and
    all(["Accepted", "ResolvedRefs"][]; . as $t |
      any($p.conditions[]?; .type == $t and .status == "True" and .observedGeneration == $g)))' <<<"$ROUTE" >/dev/null

# This focused check supports ordinary Service backends, including cross-namespace references.
jq -e '[.spec.rules[]?.backendRefs[]?] as $b | ($b | length) > 0 and
  all($b[]; (.group // "") == "" and (.kind // "Service") == "Service")' <<<"$ROUTE" >/dev/null
BACKENDS=$(jq -r --arg ns "$NS" '[.spec.rules[]?.backendRefs[]? |
  [(.namespace // $ns), .name]] | unique[] | @tsv' <<<"$ROUTE")
while IFS=$'\t' read -r BACKEND_NS SERVICE; do
  kubectl get service "$SERVICE" -n "$BACKEND_NS" -o json >/dev/null
  kubectl get endpointslices -n "$BACKEND_NS" -l "kubernetes.io/service-name=$SERVICE" -o json |
    jq -e 'any(.items[]?.endpoints[]?; .conditions.ready != false and (.addresses | length) > 0)' >/dev/null
done <<<"$BACKENDS"

ADDRESS=$(jq -er '.status.addresses[0].value | select(length > 0)' <<<"$GW")
[[ "$ADDRESS" == *:* ]] && ADDRESS="[$ADDRESS]"
# --connect-to changes the destination while retaining Host, TLS SNI and certificate checks.
HTTP_CODE=$(curl --silent --show-error --output /dev/null --write-out '%{http_code}' \
  --connect-to "${HOST}:${PORT}:${ADDRESS}:${PORT}" \
  --max-time 10 "${SCHEME}://${HOST}:${PORT}${PROBE_PATH}")
[[ "$HTTP_CODE" =~ ^[23][0-9][0-9]$ ]] || { echo "Probe failed: HTTP $HTTP_CODE" >&2; exit 1; }
echo "Selected parent accepted/resolved; Gateway programmed; ready backends; probe HTTP $HTTP_CODE"

Usage example:

chmod +x validate-httproute.sh
./validate-httproute.sh production api-route infra production-gateway https api.example.com /api/v1/health

5. Troubleshooting​

5.1 Common Issues and Resolutions​

Common issues and solutions
Gateway API Troubleshooting Guide
SymptomCauseResolution
HTTPRoute Accepted=FalseThe selected Gateway/listener has not accepted the Route attachment1. Check parentRefs and listener 2. Check hostname intersection 3. Check allowedRoutes namespace policy
HTTPRoute ResolvedRefs=FalseBackend references cannot be resolved for this parent1. Check Service name, port, and kind 2. Check ReferenceGrant for cross-namespace references 3. Inspect condition reason/message
Gateway Programmed=FalseGateway dataplane configuration is not ready1. Check Gateway and listener conditions 2. Check controller logs and LoadBalancer status 3. Check TLS listener ResolvedRefs and certificate Secret
503 Service UnavailableNo backend endpoints1. Check Service Endpoints 2. Verify Pod selector match 3. Check Pod status (Ready)
TLS Certificate ErrorInvalid Secret1. Verify Secret type kubernetes.io/tls 2. Check tls.crt, tls.key exist 3. Verify certificate validity
404 Not FoundRoute matching failure1. Check PathPrefix vs Exact type 2. Check case sensitivity 3. Check URL encoding
No Gateway AddressLoadBalancer creation failed1. Check cloud provider quota 2. Check subnet IP exhaustion 3. Check annotation typos

5.2 Debugging Commands by Controller​

AWS Load Balancer Controller

# Inspect controller logs
kubectl logs -n kube-system deployment/aws-load-balancer-controller --tail=100 -f

# Identify the Gateway's underlying NLB
kubectl get gateway <name> -n <namespace> -o jsonpath='{.metadata.annotations.service\.beta\.kubernetes\.io/aws-load-balancer-name}'

# Check NLB target group health
aws elbv2 describe-target-health --target-group-arn <arn>

# Inspect HTTPRoute events
kubectl describe httproute <name> -n <namespace>

Cilium Gateway API

# Cilium Operator logs
kubectl logs -n kube-system deployment/cilium-operator --tail=100 -f

# Dump the Envoy configuration
kubectl exec -n kube-system ds/cilium -- cilium envoy config dump > envoy-config.json

# Inspect the HTTPRoute routing table
kubectl exec -n kube-system ds/cilium -- cilium service list

# Monitor Gateway-related flows
hubble observe --protocol http --port 443

# Check Gateway status
cilium status --wait

NGINX Gateway Fabric

# NGINX Gateway logs
kubectl logs -n nginx-gateway deployment/nginx-gateway --tail=100 -f

# Inspect the NGINX configuration
kubectl exec -n nginx-gateway deployment/nginx-gateway -- nginx -T

# Check HTTPRoute mappings
kubectl describe httproute <name> -n <namespace>

# Follow access logs in real time
kubectl logs -n nginx-gateway deployment/nginx-gateway -f | grep "HTTP/1.1"

Envoy Gateway

# Envoy Gateway controller logs
kubectl logs -n envoy-gateway-system deployment/envoy-gateway --tail=100 -f

# Envoy Proxy logs (data plane)
kubectl logs -n envoy-gateway-system deployment/envoy-<gateway-name> --tail=100 -f

# Forward the Envoy admin interface port
kubectl port-forward -n envoy-gateway-system deployment/envoy-<gateway-name> 19000:19000

# Open http://localhost:19000 in a browser to inspect stats and configuration

# Dump the xDS configuration
curl http://localhost:19000/config_dump > envoy-xds-config.json

Common debugging commands

# Inspect detailed Gateway status
kubectl get gateway <name> -n <namespace> -o yaml

# Inspect detailed HTTPRoute status
kubectl get httproute <name> -n <namespace> -o yaml

# Check backend Service endpoints
kubectl get endpoints <service-name> -n <namespace>

# Check whether Pods are Ready
kubectl get pods -n <namespace> -l <selector>

# Inspect network policies for traffic blocking
kubectl get networkpolicies -n <namespace>

# Inspect events from the last 10 minutes
kubectl get events -n <namespace> --sort-by='.lastTimestamp' | tail -20