Skip to main content

CNI Configuration and Pod CIDR Routing

Published 2026-08-257 min read

Overview​

Hybrid nodes remain in the NotReady state until a CNI is running, and the Amazon VPC CNI is incompatible with hybrid nodes. Selecting a CNI and deciding on a Pod CIDR routing method are therefore prerequisites for the cluster to serve workloads. This document covers CNI selection criteria, the core Cilium installation settings (affinity and IPAM), and the configuration procedures for BGP and static routing — two of the three Pod CIDR routing methods. The third method, the Hybrid Nodes Gateway, is covered in a separate chapter.

CNI Selection Criteria​

"Cilium is the AWS-supported Container Networking Interface (CNI) for Amazon EKS Hybrid Nodes. You must install a CNI for hybrid nodes to become ready to serve workloads. Hybrid nodes appear with status Not Ready until a CNI is running." — Configure CNI for hybrid nodes

CNIHybrid node supportNotes
Amazon VPC CNIIncompatibleConfigured by default with anti-affinity for the eks.amazonaws.com/compute-type: hybrid label, so it is never scheduled on hybrid nodes
CiliumAWS-supported CNILifecycle-managed with Helm using the EKS distribution (public.ecr.aws/eks). Note that Cilium running on cloud nodes is not covered by AWS support
CalicoCommunity pathThe Calico guidance in the official User Guide has been moved to the EKS Hybrid Examples repository

Two constraints matter for this decision.

  • Gateway dependency: The Hybrid Nodes Gateway is exclusive to Cilium's VTEP feature. Choosing Calico excludes the Gateway option, limiting Pod CIDR routing to BGP or static routing (Architecture Decision Guide).
  • Mixed-mode placement isolation: Cilium must be placed exclusively on hybrid nodes and the VPC CNI exclusively on cloud nodes. The VPC CNI's anti-affinity comes built in, but on the Cilium side you must declare affinity for the hybrid label at install time to prevent it from encroaching on cloud nodes.

Core Cilium Installation Settings​

In the Cilium Helm values, the two settings specific to hybrid environments are affinity and IPAM.

# cilium-values.yaml
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: eks.amazonaws.com/compute-type
              operator: In
              values:
                - hybrid            # Schedule only on hybrid nodes
ipam:
  mode: cluster-pool
  operator:
    clusterPoolIPv4MaskSize: 26      # Per-node Pod IP slice (/26 = 64 addresses)
    clusterPoolIPv4PodCIDRList:
      - 10.86.0.0/16                 # Must match RemotePodNetwork
operator:
  unmanagedPodWatcher:
    restart: false                   # Prevent restarts of cloud-node Pods (managed by the VPC CNI)
helm install cilium oci://public.ecr.aws/eks/cilium/cilium \
  --version CILIUM_VERSION \
  --namespace kube-system \
  --values cilium-values.yaml
  • clusterPoolIPv4PodCIDRList must match the RemotePodNetwork specified at cluster creation, and since changing it during operation is effectively impossible, allocate generously from the start (Proactive Pod CIDR Allocation).
  • clusterPoolIPv4MaskSize correlates with the maximum Pod count per node (/26 = 64 IPs, /25 = 128 IPs). Verify consistency with the kubelet maxPods setting.
  • In mixed-mode clusters, unmanagedPodWatcher.restart: false is a safeguard that keeps the Cilium operator from restarting cloud-node Pods under VPC CNI management.

Choosing a Pod CIDR Routing Method​

In environments that require Pod-level inbound traffic (feature table), there are three ways to make the Pod CIDR reachable.

MethodSuitable environmentRequirementsCharacteristics
BGP dynamic routing (recommended)Medium-to-large environments with frequent node changesBGP-capable router, ASN/peering coordinationAutomatically advertises each node's Pod CIDR slice — no router changes needed when scaling out
Static routingSmall environments with a fixed node count, or equipment without BGP supportManually register per-node routes on the routerRouter routes must be updated on every node addition or replacement — operational burden accumulates
Hybrid Nodes GatewayEnvironments where routing coordination is impractical due to organizational separation or IPAM exhaustionCilium only, 2 gateway EC2 instancesEliminates on-premises routing coordination altogether — setup procedure

Per-node Pod CIDR slices are assigned dynamically by the CNI, so static routing requires identifying slice assignments in advance, registering routes per node, and re-verifying them when nodes are replaced. This management burden is what confines static routing to small environments.

Cilium BGP Control Plane Configuration​

Cilium's BGP capability (BGP Control Plane) advertises Pod CIDRs and Service addresses to the on-premises network. Configuration proceeds in three steps: ① enable via Helm → ② apply the three CRDs → ③ verify peering.

1. Enable the BGP Control Plane​

helm upgrade cilium oci://public.ecr.aws/eks/cilium/cilium \
  --namespace kube-system \
  --reuse-values \
  --set operator.rollOutPods=true \
  --set bgpControlPlane.enabled=true

Adding BGP to an existing deployment requires restarting the Cilium operator; operator.rollOutPods=true performs this as part of the Helm upgrade.

2. Configure the Three BGP CRDs​

CRDRoleKey fields
CiliumBGPClusterConfigDefines BGP instances and peers for a node grouplocalASN (node side), peerASN / peerAddress (on-premises router)
CiliumBGPPeerConfigPeer session parametersholdTimeSeconds (default 90s), keepAliveTimeSeconds (default 30s), graceful restart (default 120s) — must match the router-side settings
CiliumBGPAdvertisementDeclares what to advertiseadvertisementType: PodCIDR (Pod ranges) or Service (LB addresses)
apiVersion: cilium.io/v2alpha1
kind: CiliumBGPClusterConfig
metadata:
  name: cilium-bgp
spec:
  nodeSelector:
    matchExpressions:
      - key: eks.amazonaws.com/compute-type
        operator: In
        values:
          - hybrid
  bgpInstances:
    - name: "rack0"
      localASN: NODES_ASN
      peers:
        - name: "onprem-router"
          peerASN: ONPREM_ROUTER_ASN
          peerAddress: ONPREM_ROUTER_IP
          peerConfigRef:
            name: "cilium-peer"
---
apiVersion: cilium.io/v2alpha1
kind: CiliumBGPAdvertisement
metadata:
  name: bgp-advertisement-pods
  labels:
    advertise: bgp
spec:
  advertisements:
    - advertisementType: "PodCIDR"

The Cilium agent on each hybrid node peers individually with the on-premises router and advertises only the Pod CIDR slice it owns. Configuring the on-premises router as a BGP Route Reflector lets the router learn Pod CIDRs dynamically without participating in the data path.

3. Verify Peering​

# Confirm the Session State is established
cilium bgp peers

# Check the routes advertised per node (one slice per node)
cilium bgp routes

If the session does not establish, check the router-side consistency of ASN, peer IP, and timer values, along with whether the firewall allows TCP 179. Include Cilium's own ports (BGP TCP 179, VXLAN UDP 8472, health TCP 4240) in the Zone E rules of the firewall pre-registration request.

Summary of Recommendations​

  • Design new deployments around the EKS distribution of Cilium. Calico is a community path and excludes the Gateway option.
  • In mixed mode, always declare hybrid-label affinity in the Cilium values to keep it off cloud nodes.
  • Match clusterPoolIPv4PodCIDRList to RemotePodNetwork, and allocate generously from the start given its immutability constraint.
  • For Pod CIDR routing, evaluate BGP first; use the Gateway when routing coordination is impractical, and reserve static routing for small, fixed environments.
  • Coordinate BGP timers and ASNs with the network team in advance, and include TCP 179 in the firewall request.

References​

Official Documentation​

Technical Blogs​