Engineering Playbook · Deep Dive Session · 2026 E presenter · S notes · Q QR · P pdf eBPF DEEP DIVE 01 / 39
Engineering Playbook · Deep Dive Session

eBPF —
Programming the Kernel

No reboot required. Restricted programs verified before running inside the kernel — dissected in three chapters, from the virtual machine internals to production on EKS.

01 · CORE INTERNALS 02 · OBSERVABILITY & SECURITY 03 · eBPF ON EKS
Agenda

Three chapters, one question —
"What is happening inside the kernel?"

01

Core Internals

eBPF VM · Verifier · JIT · Maps · CO-RE/BTF · Ring Buffer — grounding the safety guarantees at the kernel-source level

FOUNDATION
02

Observability & Security

Product landscape · memleak · Off-CPU · CPU throttling · Falco vs Tetragon — how eBPF is used in the field

PRACTICE
03

eBPF on EKS

VPC CNI NetworkPolicy · Cilium ENI · Hubble · reported benchmarks — EKS components using eBPF when enabled

AWS
01
Chapter one

A virtual machine inside the kernel

origins · vm · verifier · jit · maps · co-re

The problem

Two ways to extend the kernel

The traditional way to put code in the kernel is a kernel module — powerful, but its fate is tied to the kernel's.

Kernel module (LKM · Loadable Kernel Module)

  • One bug = kernel panic, whole system down
  • Recompile & revalidate for every kernel version — no ABI (Application Binary Interface) guarantee
  • Signing, distribution, audit burden — a hard sell to the security team
  • Unrestricted memory access — no isolation whatsoever

eBPF program

  • The Verifier checks safety constraints before load — verifier checks reduce risk but do not eliminate kernel bugs
  • Runs in a register-based restricted instruction model, JIT (Just-In-Time) compiled to native speed
  • CO-RE (Compile Once – Run Everywhere) adapts to compatible kernel layouts — required hooks and features must still exist
  • Attach / detach instantly — no reboot, no module load
Instrumentation faces the same dilemma — ptrace/strace stop the target and round-trip a context switch per event; untenable in production "Program the kernel safely" — the prototype of this design existed 30 years ago
Prior art — 1992

The prototype answer — tcpdump's packet filter

tcpdump does not run its filter in userspace — it compiles the expression to bytecode, pushes it down into the kernel, and gets back only the packets that match.

USER SPACE tcpdump 'port 443' expression → BPF bytecode Receives matches only copy cost minimized KERNEL NIC full packet stream BPF interpreter runs the filter on every packet inject bytecode copy matches only no match — dropped

Three things eBPF inherited

  • Move code to where the data is — inject the filter into the kernel instead of copying every packet out to filter it
  • A verifiable, restricted instruction set — bytecode, not arbitrary machine code
  • Programmability without kernel changes — an in-kernel engine does the execution
  • eBPF keeps these principles intact and redesigns only the engine — the 'e' is for extended
cBPF (classic Berkeley Packet Filter) — 2 registers · 32-bit · sockets only → eBPF (extended BPF) — 11 registers · 64-bit · kernel-wide To this day, tcpdump filters are translated to eBPF and executed in the kernel
Evolution — 1992 to today

From packet filter to general-purpose kernel VM

The backdrop: a kernel feature takes years to travel from mainline to your distro — the kernel release cycle could not keep up with container-era networking, observability, and security demands.

1992 · BPF tcpdump packet filter 2 registers · 32-bit interpreter 2014 · eBPF — kernel 3.18 64-bit · 11 registers · Maps bpf() syscall — general-purpose VM 2016 · XDP — 4.8 eXpress Data Path NIC-level processing · Cilium debuts 2019 · BTF · CO-RE BPF Type Format portability solved · bounded loops 5.3 2020 · LSM 5.7 · Ringbuf 5.8 Linux Security Module — blocking from observing to enforcing 2021+ · Foundation eBPF Foundation founded Windows eBPF — de facto standard
Kernel release cycle ≫ the pace at which infra requirements change Modules are dangerous; forking the kernel is a maintenance sink Prior art DTrace was CDDL-licensed — un-mergeable into the GPL kernel, forcing an independent path The answer — build a safe programming layer into the kernel itself
The concept

The core idea — event-driven programs reacting to kernel events

USER SPACE Control application Go · Rust · C — libbpf Data consumers CLI · agents · monitoring KERNEL Event sources syscalls · network packets function entry/exit · scheduler perf events · LSM hooks eBPF program attached to a hook — not resident, runs only when the event fires Maps · Ring Buffer kernel ↔ userspace shared state · event stream bpf() — load · attach trigger read · write polling · stream consumption
① Event-driven — not a daemon, but a code fragment reacting to kernel events ② Sandboxed — only Verifier-proven programs ever run ③ Shared state — all data exchange goes through Maps
"Virtual machine"?

What the 'VM' really is — like the JVM, not VMware

The eBPF VM is not hardware virtualization — it is an instruction execution model defined by the kernel. The bytecode intermediate representation is what makes static analysis by the Verifier possible.

Hardware VM — KVM · VMwareeBPF VM
What is virtualizedEntire CPU · memory · devicesAn instruction-set spec — 11 registers · 64-bit · 512 B stack
Reason to existOS-level isolationA verifiable intermediate representation + architecture independence (x86 · ARM · RISC-V)
ExecutionVia hypervisorJIT-compiled → resides in the kernel as native machine code
Runtime overheadAlways presentJIT avoids interpretation; hooks, helpers, maps and the program still consume CPU
Four reasons it earns the name "VM" ① A dedicated bytecode + 11-register execution layer ② Built-in JIT — virtual instructions to native machine code ③ Verifier — a sandbox that proves safety before execution ④ Maps and Helpers/KFuncs are the only doors — arbitrary kernel memory access is blocked
Load-time costs: Verifier analysis + JIT compilation; runtime cost depends on the program and hook Runtime limits constrain expressiveness, not speed — the 512 B stack and instruction cap are the price of the proof
Register machine

11 registers — designed to map 1:1 onto hardware

Basic instructions are 8 bytes; wide instructions use 16 bytes; registers correspond directly to x86_64/ARM64 registers — which is why the JIT emits native machine code with no extra translation layer.

R0

Return value

Holds the program's exit code and the result of helper function calls

RETURN
R1–R5

Argument passing

Helper call arguments — on entry, the context pointer arrives in R1

ARGUMENTS
R6–R9

Callee-saved

Registers whose values survive across helper calls — home for local state

PRESERVED
R10

Read-only frame pointer

Points to the dedicated 512 B stack and is non-writable — kernel-memory corruption via stack-pointer manipulation is ruled out by design

READ-ONLY
Compact instructions + architecture-specific register mapping — minimal JIT translation cost is baked into the design Profiling can identify expensive code paths — quantify improvements with workload-specific measurements
From C to kernel

From source to kernel — the execution pipeline

BPF C source prog.bpf.c Clang / LLVM -target bpf Bytecode (.o) ELF + .BTF bpf() syscall BPF_PROG_LOAD Verifier the safety-proof gate JIT compiler native machine code Hook attach kprobe · tc · xdp · lsm Load rejected -EACCES on verification failure — the program never sets foot in the kernel
11 registers · R0–R10 R10 = read-only frame pointer 512 B stack Nothing runs without passing the Verifier
The gatekeeper

The Verifier — proving safety before execution

Not runtime isolation but load-time static verification. If the proof fails, the kernel refuses the program.

What it checks

  • Exhaustive CFG (Control Flow Graph) walk — simulates every branch path; one unprovable path rejects the whole program
  • Termination proof — statically guarantees the program finishes; infinite loops are impossible
  • Register & stack state tracking — scalar/pointer types and value ranges, including types of values spilled to the stack
  • Pointer arithmetic bounds checking — rejects out-of-map access, null dereference, uninitialized reads
  • Reference tracking — a ringbuf reserve must end in submit/discard; kernel resource leaks are cut off

Limits and evolution

  • Verification complexity cap — a 1 million instruction exploration budget
  • Loops: banned at first → bounded loops in kernel 5.3+, plus the bpf_loop() helper
  • 512 B stack — use maps as scratch space for large buffers
  • Conservative by principle — safe-but-unprovable still gets rejected
Where programs attach

Program types and hooks — observation points across the kernel

User space uprobe USDT Syscall boundary tracepoint LSM seccomp Kernel core kprobe fentry / fexit perf_event Network stack socket filter sock_ops tc (ingress/egress) NIC driver XDP ← before the packet touches the stack — the earliest point
Shared state

Maps — shared state between kernel and userspace

Persistent state is commonly stored in maps; global variables are also backed by maps, and all kernel functionality is reached through helpers and kfuncs.

Key map types

  • HASH / ARRAY — general key-value; LRU (Least Recently Used) variants evict automatically
  • PERCPU_* — per-CPU copies for lock-free, high-speed aggregation
  • STACK_TRACE — the call-stack store populated by bpf_get_stackid()
  • RINGBUF — the standard event stream since kernel v5.8+

Doors into kernel functionality

  • Helper functions — a stable, fixed-ABI API (bpf_map_lookup_elem, bpf_ktime_get_ns …)
  • KFuncs — kernel functions exposed directly; flexible but no stability guarantee
  • Helpers grow slowly; new capabilities tend to ship as kfuncs first
  • Each program type has a different callable helper set — enforced by the Verifier
Data structure primer

The ring buffer — a bounded queue from a fixed array and two pointers

eBPF did not invent this — printk/dmesg logs · NIC RX/TX rings · ftrace · perf · io_uring: a data structure proven for decades all over the kernel.

Fixed N slots — O(1) read & write · no runtime allocation C — read P — write 0 1 2 3 4 5 6 7 8 9 10 11 wrap at the end — index = pos & (N−1) · size forced to 2ᵏ P == C empty between C and P data to read C right ahead of P full — policy decision

Why the kernel loves this structure

  • No malloc on hot paths like interrupts or the scheduler — only writes into pre-allocated slots
  • Memory usage has a ceiling guaranteed at design time — even a flood stops at the buffer size
  • Decouples ns-scale producers from ms-scale consumers — the buffer absorbs the gap
  • One contiguous memory block — shared with userspace via mmap, read without copying
The full-buffer policy defines the use case — logs (dmesg) overwrite the oldest; event pipes can reject new records; programs must account for failed ringbuf reservations Perf Buffer and BPF Ring Buffer both descend from this structure — what changed is the next four slides
Legacy path — how it worked

The old path — per-CPU rings: assemble → copy → threshold-based notification

Perf Buffer takes the ring from the last slide and puts one on every CPU — each BPF program writes only to its own CPU's ring, and the consumer reads and merges as many rings as there are CPUs.

KERNEL SPACE CPU 0 — BPF program ① assemble record on stack ② copy full — new events lost CPU 1 — BPF program ① assemble record on stack ② copy load varies per CPU CPU 2 — BPF program ① assemble record on stack ② copy near-empty — reserved memory wasted USER SPACE Userspace consumer epoll on fd × CPU count — reads each ring separately; time-ordering is the consumer's job ③ event-count / byte-watermark notification — tune thresholds to batch wakeups
The ring itself is unchanged from the last slide — what differs is the arrangement: one per CPU, records assembled then copied in Four costs fall straight out of this picture (next slide)
Legacy — perf buffer

Legacy Perf Buffer — the price of per-CPU buffers

The legacy channel for shipping events from kernel to userspace was an independent buffer per CPU — all four limitations derive from that single design choice.

01

Memory over-allocation

Reserves CPU count × buffer size — one buffer overflows while another sits empty, wasted either way

OVER-ALLOC
02

Event reordering

No ordering across CPUs — if a process migrates, fork → exec can arrive reversed

REORDER
03

Late loss detection

Out-of-space is only discovered when copying the record built on the stack — the assembly cost is already paid

LATE FAIL
04

Wakeup storms

Notifications use event-count or byte thresholds; low thresholds can increase wakeup overhead

SIGNAL STORM
The fix converges to one move — merge the buffers into a single global one (next slide)
New path — how it works

The new path — reserve → write in place → commit, notify only when needed

One global ring — every CPU claims space first with reserve(), writes directly into it, then publishes with submit(). The copy disappears, and ordering is fixed by reservation order.

KERNEL SPACE CPU 0 BPF program CPU 1 BPF program CPU 2 BPF program ① reserve() — claims a slot only · fails immediately if space is short BUSY ② write in place — no copy ③ submit() — clearing the busy bit = published free space double mapping same phys pages ×2 mapping 1 mapping 2 record across the boundary — read contiguously, no split USER SPACE ④ adaptive notification — notify when the consumer has caught up Userspace consumer one fd — direct reads via mmap · reservation order within one ring
The consumer stops at a BUSY record — only committed records are visible, in reservation order Single ring · reserve · in-place writes · adaptive notification — what each one fixes is the next slide
Kernel → userspace

BPF Ring Buffer — a shared-buffer alternative (v5.8+)

An MPSC (Multi-Producer Single-Consumer) design where all CPUs share one buffer — answering Perf Buffer's four limitations with four mechanisms.

MPSC

One global shared buffer

Every CPU writes to one buffer; userspace reads it — reservation ordering within one ring and memory efficiency at once.

ORDER + MEMORY
RESERVE / SUBMIT

Two-phase zero-copy

reserve() claims space first — if short, it fails immediately so the program can react. Write directly into buffer memory, then submit() — the intermediate copy is gone.

§11 REF TRACKING
ADAPTIVE

Adaptive notification

Notify if the consumer has caught up to the committed record; otherwise it will reach the record without another wakeup.

NO SIGNAL STORM
DOUBLE MAP

Virtual-memory double mapping

The same physical pages are mapped twice, back to back — records straddling the boundary read contiguously, no wrap-around handling.

ZERO WRAP COST
Every reserve must end in submit/discard — enforced by the Verifier's reference tracking (§11) Reordering · over-allocation · late loss detection · wakeup storms — all resolved by the single-buffer design
Side by side

Ring buffer generations — same name, different design

Perf Buffer is a ring buffer inside, too — what the eBPF Ring Buffer changed is not the ring itself but everything about how rings are placed, written, and read.

Legacy ring — Perf BufferBPF Ring Buffer (v5.8+)
LayoutIndependent ring per CPU — as many rings as CPUsOne global ring — MPSC, shared by all CPUs
Write pathAssemble record on the stack, then copy into the bufferreserve() → write directly into the buffer → submit() — zero-copy
Out of spaceDiscovered only at copy time — assembly cost already paidreserve() fails immediately — the program can react
Event orderingNo cross-CPU guarantee — fork → exec can reverseSingle buffer — reservation order within one ring
Boundary handlingRecords split at the boundary — read twice and reassembleVirtual-memory double mapping — contiguous read, no split
Consumer notificationConfigurable event-count / byte thresholdsAdaptive — notify when the consumer has caught up
Consumer APIs look alike — perf_buffer__poll ↔ ring_buffer__poll; migration is cheap Choose by ordering, contention and compatibility requirements — per-CPU Perf Buffers remain useful
Compile once — run everywhere

CO-RE — compile once, run everywhere

The root problem: kernel struct offsets differ across versions and configs. CO-RE resolves relocations at load time using target BTF.

COMPILE TIME · DEV MACHINE BPF C source includes vmlinux.h Clang -g emits relocation records .o (ELF) .BTF + .BTF.ext "which fields does it touch" LOAD TIME · TARGET NODE The kernel's own BTF /sys/kernel/btf/vmlinux libbpf recompute offsets · patch instructions Kernel load offsets fit this kernel
BTF dedup in 5 stages — strings → non-reference types → struct/union canonicalization → reference types → compaction; hundreds of MB of DWARF down to a few MB Measured — x86_64 pt_regs offsets (di=112 · si=104 · dx=96) patched into the bytecode at load time Slim static binaries with no runtime compilation — unlike BCC (BPF Compiler Collection) embedding clang
The whole picture

The integrated flow — where the pieces interlock

Overlaying everything so far: the load path (compile → verify → JIT) and the runtime channels (Helpers · Maps · Ring Buffer) on one slide.

USER SPACE Clang / LLVM §10 · .o + .BTF.ext libbpf loader §20 · offset patching Kernel BTF /sys/kernel/btf/vmlinux the kernel's type dictionary Userspace consumer §18 · Ring Buffer reads · map polling KERNEL SPACE bpf() syscall BPF_PROG_LOAD Verifier §11 · exhaustive CFG walk reference tracking JIT §10 · native machine code BPF VM execution §13 · Helpers / KFuncs · Maps MPSC Ring Buffer §18 · reserve / submit · adaptive notification
Top row is the load path, bottom row the runtime channels — § numbers point to the detail slides Without patching the kernel — verified bytecode lives inside it
The frontier

Recent features pushing the boundary

Privilege delegation, large memory, userspace execution — eBPF's constraints are being lifted one by one.

KERNEL v6.9+

BPF Token

Instead of granting all of CAP_BPF, issue a delegation token — a container inside a user namespace loads only the allowed program and map types. Least privilege for multi-tenant environments.

DELEGATION
KERNEL v6.9+

BPF Arena

A large memory region where kernel and userspace share the same pages. Enables pointer-based data structures (trees, lists) instead of the map API.

SHARED MEMORY
EXPERIMENTAL

bpftime

Kernel uprobes trap via int3 with a heavy kernel round-trip — bpftime instruments supported probes in userspace. Performance and compatibility depend on the workload.

USERSPACE RUNTIME
02
Chapter two

Observe and defend

landscape · memleak · off-cpu · throttling · falco vs tetragon · tuning

Tooling

The tool landscape — a ladder of abstraction

↑

bpftrace

DSL (Domain-Specific Language) one-liners — the standard for ad-hoc diagnosis. Use like awk, then throw away. bpftrace -e 'tracepoint:syscalls:sys_enter_openat { @[comm] = count(); }'

AD-HOC
·

BCC

Python + C framework with proven tools like memleak and offcputime. Caveat: runtime clang compilation — heavy to ship.

TOOLBOX
·

libbpf + CO-RE

The production standard — slim static binaries, no per-kernel recompilation. Cilium, Tetragon, and most commercial agents ship this way.

PRODUCTION
↓

bpftool

The standard CLI to inspect and dump programs, maps, and BTF loaded in the kernel. The answer to "what is attached to my kernel right now?"

INTROSPECTION
← quick experimentsproduction rollout →
Ecosystem

Product landscape — observe · detect · enforce

Products on the same eBPF stand in different places — how far they see, and where they start blocking, is the axis.

OBSERVE

Observability · profiling

Pixie — auto-instruments HTTP/gRPC/SQL with zero code change, builds service maps. Parca — always-on low-rate stack sampling, continuous profiling. Hubble — sidecar-free L3–L7 visibility (§36).

NO-INSTRUMENT
DETECT

Runtime detection

Falco — rule-based syscall behavior detection with configurable rules (§29). Tracee — Aqua Security; forensics-oriented detection with precise kernel event capture.

ALERT-ONLY
ENFORCE

In-kernel blocking

Tetragon — synchronous verdicts and blocking at LSM hooks (§30). KubeArmor — workload hardening that pre-empts file/process/socket access via BPF-LSM and AppArmor.

IN-KERNEL BLOCK
CORRELATE

Multi-dimensional — CADR

ARMO and peers — correlate kernel eBPF signals with cloud APIs, the K8s control plane, and app code; reconstruct attack stories instead of isolated alerts.

ATTACK STORY
The classification axis — does it ship events to userspace to 'report', or does it 'decide' inside the kernel? The structural gap between detect ↔ enforce is dissected in §29–30
Case study № 1

Tracking memory leaks — all aggregation stays in the kernel

How it works

  • Probes on the allocation paths — uprobes for user malloc·calloc·realloc·mmap, kprobes for kernel kmalloc → record address → (stack ID · size · timestamp) in a map
  • On free, delete the entry — long-lived entries = leak candidates
  • bpf_get_stackid() stores the call stack in a STACK_TRACE map
  • Userspace only reads the map periodically — no event streaming, minimal overhead
# track outstanding allocations of PID 1234 every 10 s $ memleak -p 1234 10 [13:42:01] Top 10 stacks with outstanding allocations: 524288 bytes in 128 allocations from stack malloc+0x1f grow_request_buffer+0x42 handle_connection+0x8d start_thread+0xd9 # --older: allocation candidates still outstanding after 60 s · omit -p to trace kernel leaks $ memleak -p 1234 --older 60000
Unlike Valgrind, no restart needed — attaches dynamically to a running process; in-kernel aggregation, no CPU emulation The cost of uprobes — an int3 breakpoint trap round-trips the kernel per call; at high malloc rates the trap itself is the bottleneck The workaround removes kernel entry — bpftime runs the probe as a userspace inline hook, with workload-dependent overhead (§22)
Case study № 2

Off-CPU analysis — seeing the time spent off the CPU

Task A timeline ON-CPU BLOCKED — locks · I/O waits the profiler's blind spot ON-CPU kprobe: finish_task_switch() the single point every context switch passes through record t₀ aggregate Δt Δt accumulated in a per-stack histogram map — tens of thousands of events per second digested in-kernel

Why it matters

  • On-CPU profiles see only half the story — waiting time never shows up in samples
  • Lock contention, disk I/O, page faults are often the real latency culprits
  • Record switch-out time → on return, aggregate Δt per stack, in-kernel
  • The deliverable is an Off-CPU Flame Graph — which stacks waited, and for how long
Case study № 3

Diagnosing CPU throttling — runnable but not allowed to run

CFS (Completely Fair Scheduler) Bandwidth Control — limits.cpu → quota / 100 ms period Running 40 ms quota spent THROTTLED — forced 60 ms wait runnable but unschedulable — whole cgroup Next period refill → resume runqlat — wait distribution from runnable to running normally tens of µs → skews right with 100 ms-class spikes under throttling cross-check scheduler delay with cpu.stat throttling counters and node CPU pressure

Why the usual metrics miss it

  • The CPU utilization graph sits flat against the limit — busy and strangled look identical
  • cAdvisor throttle counters give counts and totals only — no idea which code path got delayed by how much
  • runqlat spikes + Off-CPU stacks (§27) showing no blocking traces (no io_schedule, no futex_wait) → investigate CPU contention or quota throttling; confirm with cpu.stat throttle deltas
  • The cross-reading of the two tools is your quantitative basis for resizing limits
Observed combinationDiagnosisPrescription
runqlat spikes + no blocking traces in stacks — awake but can't runCPU contention or quota throttling — correlate cpu.statRaise quota only after confirming throttling and available node CPU
runqlat normal + stacks ending in io_schedule/futex_wait — went to sleep voluntarilyI/O / lock bottleneck — unrelated to CPURaising limits does nothing — fix storage or lock design
Runtime security № 1

Falco — syscall detection and alerts

Process execve("/bin/sh") tracepoint sys_enter · argument snapshot Ring Buffer event stream Userspace rule engine verdict: "shell spawned in a container!" asynchronous detection does not guarantee blocking before the syscall completes time of argument check ≠ time of use → TOCTOU (Time-of-Check to Time-of-Use) race an asynchronous pipeline — detection only
Rule-based detection with ecosystem integrations A later response cannot guarantee prevention of the triggering operation
Runtime security № 2

Tetragon — decides in the kernel, blocks on the spot

Process execve("/bin/sh") LSM Hook called after args are final — no TOCTOU In-kernel Policy TracingPolicy CRD → BPF synchronous verdict Allow emit an event log only Block LSM return value → denial other actions depend on hook type no userspace round-trip — verdict and enforcement complete in the same kernel context
Falco = after-the-fact detection Tetragon = synchronous execution blocking (enforcement) Kubernetes-aware — per-Pod/namespace policies via the TracingPolicy CRD (Custom Resource Definition) The next step is correlation — weaving kernel eBPF data with cloud API and app signals into full attack stories (ARMO CADR · Cloud Application Detection & Response)
Operations

Production checklist

01

JIT settings

net.core.bpf_jit_enable=1 — the default on most modern distros. bpf_jit_harden blinds constants against JIT spraying but costs performance — choose by trust boundary.

SYSCTL
02

Memory accounting — a kernel-version fork

Before v5.11: map memory counts against RLIMIT_MEMLOCK — check and raise ulimit -l when the configured limit is insufficient. v5.11+: switched to cgroup memory accounting — container memory limits now include BPF maps, so size them accordingly.

MEMLOCK
03

Shipping strategy

libbpf CO-RE static binaries instead of BCC runtime compilation. For older kernels without BTF, inject external BTF from BTFHub.

CO-RE
04

Overhead management

Filter as much as possible inside the kernel — cost jumps the moment events cross into userspace. Aggregate in per-CPU maps to eliminate lock contention.

OVERHEAD
03
Chapter three

eBPF on EKS

vpc cni · cilium · hubble · benchmarks

Already in production

The EKS eBPF landscape — already in production

eBPF powers these EKS networking options when installed and enabled. Understanding it adds a debugging layer.

AWS · v1.14+

VPC CNI NetworkPolicy

When NetworkPolicy is enabled, the agent attaches eBPF programs to supported Pods' host veth. Policy enforcement with no iptables chains.

POLICY
CNCF

Cilium

An eBPF network datapath with userspace IPAM/control components. Optional kube-proxy replacement and ENI routing.

CNI
CILIUM

Hubble

L3/L4/L7 flow observability from the same eBPF datapath — service maps with zero extra instrumentation.

OBSERVABILITY
AWS MANAGED

Network Flow Monitor

An agent DaemonSet collecting TCP retransmits and RTT (Round-Trip Time) via the sock_ops hook.

MONITORING
eBPF on EKS № 1

VPC CNI NetworkPolicy — a division of labor: controller and agent

NetworkPolicy user-defined object Network Policy Controller EKS control plane · AWS-managed resolves selectors → concrete Pod IP sets PolicyEndpoints CRD — resolved results published WORKER NODE aws-network-policy-agent DaemonSet — watches PolicyEndpoints eBPF probes ingress / egress verdicts host veth Pod interface
eBPF policy enforcement — enable NetworkPolicy and check supported nodes Troubleshooting step one: inspect PolicyEndpoint resources alongside the NetworkPolicy Scope — Pod eth0 only · does not apply to host networking / Windows / Fargate
eBPF on EKS № 2

Network Flow Monitor — reading AWS's own eBPF in the source

The NFM agent is open source (Rust · Apache-2.0) — it shows exactly how chapter 1's concepts are used in production code.

HOOK

A single sock_ops program

sock_ops callbacks on cgroup v2 collect TCP connect, RTT and retransmits. The BPF_OK return does not block connections.

§12 APPLIED
MAP

Map polling instead of a Ring Buffer

Per-socket maps are aggregated every 500 ms by default. This design does not stream every event through a ring buffer.

§18 COUNTERPOINT
SAMPLE

Sampling at the entrance only

Sampling is decided at new-socket entry. Map capacity, eviction and processing errors can still limit tracking.

CONSISTENCY
CAP

Drops its own privileges

Drops CAP_SYS_ADMIN, CAP_PERFMON and CAP_NET_ADMIN after initialization; retains CAP_BPF. Other privileges depend on deployment.

LEAST PRIV
Support is version-specific — consult AWS supported Linux distributions and kernel requirements (5.8+) Ops detail (NHI · enrichment · rollout) — Playbook NFM doc
eBPF on EKS № 3

Cilium ENI mode — an eBPF datapath with no overlay

Pod VPC-native IP eBPF (tc hook) service resolution · policy · O(1) lookup ENI → VPC routing no overlay encapsulation kube-proxy optional replacement — separate setting Cilium Operator EC2 API — ENI/IP allocation CiliumNode CRD per-node IP pool Agent assigns Pod IPs

What changes

  • Optional kube-proxy replacement — configured separately from ENI IPAM; eBPF maps replace iptables Service chains
  • ENI IPAM (IP Address Management) — Pods get VPC IPs directly, overlay overhead removed
  • Hubble — enable flow observability; L7 requires protocol-aware proxy configuration
  • Optional integrations — Gateway API · mutual authentication · Bandwidth Manager; ENI mode alone does not enable them
Numbers, not vibes

eBPF in numbers — hands-on benchmarks

Reported in the playbook benchmark — 2026-02-09 · EKS 1.31 · m6i.xlarge, median of at least 3 runs. Not rerun for this review.

MetricA · VPC CNI (default)E · Cilium ENI + full tuningReading
TCP throughput12.41 Gbps12.40 GbpsSimilar measured throughput in this test; not proof of sustained instance bandwidth
Pod-to-Pod RTT p504,894 µs3,135 µseBPF datapath + tuning — ~36% lower
UDP loss (iperf3 saturation)20.4%0.03%Tuned configuration differs; the table alone cannot isolate the cause of UDP loss
HTTP p99 @1k QPS10.9 ms9.9 msThe gap narrows at the application level — the CNI is only part of the bottleneck
Conclusion: pick a CNI on latency · features · operating model — not throughput Cilium service mesh and Istio Ambient use different datapaths; both can reduce per-Pod sidecar deployment
Take these home

Read the kernel, and the cluster comes into focus

1

The Verifier is the contract

eBPF uses load-time verification, not runtime isolation. Constraints like the 512 B stack and bounded loops are the price of that proof.

2

CO-RE changed shipping

BTF relocation reduces per-kernel recompilation on compatible targets. libbpf static binaries are the production standard — keep BCC for experiments.

3

On EKS it's already table stakes

The NetworkPolicy agent, Cilium, and NFM all run on eBPF — and NFM's source is public. The moment you understand it, you gain a debugging layer.

One sentence through all three chapters — eBPF is a verified programming layer that changes the kernel's behavior without changing the kernel.

Wrap-up

The kernel is no longer a black box

Resources to go deeper — questions and discussion welcome.

Learn

eBebpf.io — What is eBPF?OVERVIEW
Kkernel.org BPF DocumentationKERNEL
BGBrendan Gregg — BPF Performance ToolsBOOK

Build on EKS

CCilium ENI ModeCNI
TTetragon — eBPF Security ObservabilitySECURITY
PBPlaybook — VPC CNI Deep DivePLAYBOOK
THANK YOU Q & A Engineering Playbook — devfloor9.github.io/engineering-playbook