Engineering Playbook · Deep Dive Session · 2026 E presenter · S notes · Q QR · P pdf eBPF DEEP DIVE 01 / 39
Engineering Playbook · Deep Dive Session

eBPF —
Programming the Kernel

No reboots. No crashes. Sandboxed programs running safely inside the kernel — dissected in three chapters, from the virtual machine internals to production on EKS.

01 · CORE INTERNALS 02 · OBSERVABILITY & SECURITY 03 · eBPF ON EKS
Agenda

Three chapters, one question —
"What is happening inside the kernel?"

01

Core Internals

eBPF VM · Verifier · JIT · Maps · CO-RE/BTF · Ring Buffer — grounding the safety guarantees at the kernel-source level

FOUNDATION
02

Observability & Security

Product landscape · memleak · Off-CPU · CPU throttling · Falco vs Tetragon — how eBPF is used in the field

PRACTICE
03

eBPF on EKS

VPC CNI NetworkPolicy · Cilium ENI · Hubble · hands-on benchmarks — EKS already runs on eBPF

AWS
01
Chapter one

A virtual machine inside the kernel

origins · vm · verifier · jit · maps · co-re

The problem

Two ways to extend the kernel

The traditional way to put code in the kernel is a kernel module — powerful, but its fate is tied to the kernel's.

Kernel module (LKM · Loadable Kernel Module)

  • One bug = kernel panic, whole system down
  • Recompile & revalidate for every kernel version — no ABI (Application Binary Interface) guarantee
  • Signing, distribution, audit burden — a hard sell to the security team
  • Unrestricted memory access — no isolation whatsoever

eBPF program

  • The Verifier proves safety before load — crashing is impossible
  • Runs in a register-based sandboxed VM, JIT (Just-In-Time) compiled to native speed
  • CO-RE (Compile Once – Run Everywhere) ports across kernel versions — no recompilation
  • Attach / detach instantly — no reboot, no module load
Instrumentation faces the same dilemma — ptrace/strace stop the target and round-trip a context switch per event; untenable in production "Program the kernel safely" — the prototype of this design existed 30 years ago
Prior art — 1992

The prototype answer — tcpdump's packet filter

tcpdump does not run its filter in userspace — it compiles the expression to bytecode, pushes it down into the kernel, and gets back only the packets that match.

USER SPACE tcpdump 'port 443' expression → BPF bytecode Receives matches only copy cost minimized KERNEL NIC full packet stream BPF interpreter runs the filter on every packet inject bytecode copy matches only no match — dropped

Three things eBPF inherited

  • Move code to where the data is — inject the filter into the kernel instead of copying every packet out to filter it
  • A verifiable, restricted instruction set — bytecode, not arbitrary machine code
  • Programmability without kernel changes — an in-kernel engine does the execution
  • eBPF keeps these principles intact and redesigns only the engine — the 'e' is for extended
cBPF (classic Berkeley Packet Filter) — 2 registers · 32-bit · sockets only → eBPF (extended BPF) — 11 registers · 64-bit · kernel-wide To this day, tcpdump filters are translated to eBPF and executed in the kernel
Evolution — 1992 to today

From packet filter to general-purpose kernel VM

The backdrop: a kernel feature takes years to travel from mainline to your distro — the kernel release cycle could not keep up with container-era networking, observability, and security demands.

1992 · BPF tcpdump packet filter 2 registers · 32-bit interpreter 2014 · eBPF — kernel 3.18 64-bit · 11 registers · Maps bpf() syscall — general-purpose VM 2016 · XDP — 4.8 eXpress Data Path NIC-level processing · Cilium debuts 2019 · BTF · CO-RE BPF Type Format portability solved · bounded loops 5.3 2020 · LSM 5.7 · Ringbuf 5.8 Linux Security Module — blocking from observing to enforcing 2021+ · Foundation eBPF Foundation founded Windows eBPF — de facto standard
Kernel release cycle ≫ the pace at which infra requirements change Modules are dangerous; forking the kernel is a maintenance sink Prior art DTrace was CDDL-licensed — un-mergeable into the GPL kernel, forcing an independent path The answer — build a safe programming layer into the kernel itself
The concept

The core idea — event-driven programs reacting to kernel events

USER SPACE Control application Go · Rust · C — libbpf Data consumers CLI · agents · monitoring KERNEL Event sources syscalls · network packets function entry/exit · scheduler perf events · LSM hooks eBPF program attached to a hook — not resident, runs only when the event fires Maps · Ring Buffer kernel ↔ userspace shared state · event stream bpf() — load · attach trigger read · write polling · stream consumption
① Event-driven — not a daemon, but a code fragment reacting to kernel events ② Sandboxed — only Verifier-proven programs ever run ③ Shared state — all data exchange goes through Maps
"Virtual machine"?

What the 'VM' really is — like the JVM, not VMware

The eBPF VM is not hardware virtualization — it is an instruction execution model defined by the kernel. The bytecode intermediate representation is what makes static analysis by the Verifier possible.

Hardware VM — KVM · VMwareeBPF VM
What is virtualizedEntire CPU · memory · devicesAn instruction-set spec — 11 registers · 64-bit · 512 B stack
Reason to existOS-level isolationA verifiable intermediate representation + architecture independence (x86 · ARM · RISC-V)
ExecutionVia hypervisorJIT-compiled → resides in the kernel as native machine code
Runtime overheadAlways presentEffectively zero — same cost as a kernel function call, no interpreter
Four reasons it earns the name "VM" ① A dedicated bytecode + 11-register execution layer ② Built-in JIT — virtual instructions to native machine code ③ Verifier — a sandbox that proves safety before execution ④ Maps and Helpers/KFuncs are the only doors — arbitrary kernel memory access is blocked
The real cost is paid once at load time — Verifier analysis + JIT compilation Runtime limits constrain expressiveness, not speed — the 512 B stack and instruction cap are the price of the proof
Register machine

11 registers — designed to map 1:1 onto hardware

Instructions are fixed 8-byte; registers correspond directly to x86_64/ARM64 registers — which is why the JIT emits native machine code with no extra translation layer.

R0

Return value

Holds the program's exit code and the result of helper function calls

RETURN
R1–R5

Argument passing

Helper call arguments — on entry, the context pointer arrives in R1

ARGUMENTS
R6–R9

Callee-saved

Registers whose values survive across helper calls — home for local state

PRESERVED
R10

Read-only frame pointer

Points to the dedicated 512 B stack and is non-writable — kernel-memory corruption via stack-pointer manipulation is ruled out by design

READ-ONLY
Fixed-length instructions + 1:1 register mapping — minimal JIT translation cost is baked into the design Proof point — Meta Strobelight: one code-line fix found via eBPF profiling saved the equivalent of 15,000 servers a year
From C to kernel

From source to kernel — the execution pipeline

BPF C source prog.bpf.c Clang / LLVM -target bpf Bytecode (.o) ELF + .BTF bpf() syscall BPF_PROG_LOAD Verifier the safety-proof gate JIT compiler native machine code Hook attach kprobe · tc · xdp · lsm Load rejected -EACCES on verification failure — the program never sets foot in the kernel
11 registers · R0–R10 R10 = read-only frame pointer 512 B stack Nothing runs without passing the Verifier
The gatekeeper

The Verifier — proving safety before execution

Not runtime isolation but load-time static verification. If the proof fails, the kernel refuses the program.

What it checks

  • Exhaustive CFG (Control Flow Graph) walk — simulates every branch path; one unprovable path rejects the whole program
  • Termination proof — statically guarantees the program finishes; infinite loops are impossible
  • Register & stack state tracking — scalar/pointer types and value ranges, including types of values spilled to the stack
  • Pointer arithmetic bounds checking — rejects out-of-map access, null dereference, uninitialized reads
  • Reference tracking — a ringbuf reserve must end in submit/discard; kernel resource leaks are cut off

Limits and evolution

  • Verification complexity cap — a 1 million instruction exploration budget
  • Loops: banned at first → bounded loops in kernel 5.3+, plus the bpf_loop() helper
  • 512 B stack — use maps as scratch space for large buffers
  • Conservative by principle — safe-but-unprovable still gets rejected
Where programs attach

Program types and hooks — observation points across the kernel

User space uprobe USDT Syscall boundary tracepoint LSM seccomp Kernel core kprobe fentry / fexit perf_event Network stack socket filter sock_ops tc (ingress/egress) NIC driver XDP ← before the packet touches the stack — the earliest point
Shared state

Maps — shared state between kernel and userspace

eBPF programs cannot hold state — all state lives in maps, and all kernel functionality is reached through helpers and kfuncs.

Key map types

  • HASH / ARRAY — general key-value; LRU (Least Recently Used) variants evict automatically
  • PERCPU_* — per-CPU copies for lock-free, high-speed aggregation
  • STACK_TRACE — the call-stack store populated by bpf_get_stackid()
  • RINGBUF — the standard event stream since kernel v5.8+

Doors into kernel functionality

  • Helper functions — a stable, fixed-ABI API (bpf_map_lookup_elem, bpf_ktime_get_ns …)
  • KFuncs — kernel functions exposed directly; flexible but no stability guarantee
  • Helpers grow slowly; new capabilities tend to ship as kfuncs first
  • Each program type has a different callable helper set — enforced by the Verifier
Data structure primer

The ring buffer — an infinite queue from a fixed array and two pointers

eBPF did not invent this — printk/dmesg logs · NIC RX/TX rings · ftrace · perf · io_uring: a data structure proven for decades all over the kernel.

Fixed N slots — O(1) read & write · no runtime allocation C — read P — write 0 1 2 3 4 5 6 7 8 9 10 11 wrap at the end — index = pos & (N−1) · size forced to 2ᵏ P == C empty between C and P data to read C right ahead of P full — policy decision

Why the kernel loves this structure

  • No malloc on hot paths like interrupts or the scheduler — only writes into pre-allocated slots
  • Memory usage has a ceiling guaranteed at design time — even a flood stops at the buffer size
  • Decouples ns-scale producers from ms-scale consumers — the buffer absorbs the gap
  • One contiguous memory block — shared with userspace via mmap, read without copying
The full-buffer policy defines the use case — logs (dmesg) overwrite the oldest; event pipes (perf · eBPF) drop the new event and count it Perf Buffer and BPF Ring Buffer both descend from this structure — what changed is the next four slides
Legacy path — how it worked

The old path — per-CPU rings: assemble → copy → notify every time

Perf Buffer takes the ring from the last slide and puts one on every CPU — each BPF program writes only to its own CPU's ring, and the consumer reads and merges as many rings as there are CPUs.

KERNEL SPACE CPU 0 — BPF program ① assemble record on stack ② copy full — new events lost CPU 1 — BPF program ① assemble record on stack ② copy load varies per CPU CPU 2 — BPF program ① assemble record on stack ② copy near-empty — reserved memory wasted USER SPACE Userspace consumer epoll on fd × CPU count — reads each ring separately; time-ordering is the consumer's job ③ unconditional notify per event — ignores consumer state · wakeup storms at high rates
The ring itself is unchanged from the last slide — what differs is the arrangement: one per CPU, records assembled then copied in Four costs fall straight out of this picture (next slide)
Legacy — perf buffer

Legacy Perf Buffer — the price of per-CPU buffers

The legacy channel for shipping events from kernel to userspace was an independent buffer per CPU — all four limitations derive from that single design choice.

01

Memory over-allocation

Reserves CPU count × buffer size — one buffer overflows while another sits empty, wasted either way

OVER-ALLOC
02

Event reordering

No ordering across CPUs — if a process migrates, fork → exec can arrive reversed

REORDER
03

Late loss detection

Out-of-space is only discovered when copying the record built on the stack — the assembly cost is already paid

LATE FAIL
04

Wakeup storms

Unconditional signal per event, regardless of consumer state — at high event rates the notification cost dominates

SIGNAL STORM
The fix converges to one move — merge the buffers into a single global one (next slide)
New path — how it works

The new path — reserve → write in place → commit, notify only when needed

One global ring — every CPU claims space first with reserve(), writes directly into it, then publishes with submit(). The copy disappears, and ordering is fixed by reservation order.

KERNEL SPACE CPU 0 BPF program CPU 1 BPF program CPU 2 BPF program ① reserve() — claims a slot only · fails immediately if space is short BUSY ② write in place — no copy ③ submit() — clearing the busy bit = published free space double mapping same phys pages ×2 mapping 1 mapping 2 record across the boundary — read contiguously, no split USER SPACE ④ adaptive notification — wake only when the consumer lags Userspace consumer one fd — direct reads via mmap · global order preserved
The consumer stops at a BUSY record — only committed records are visible, in reservation order Single ring · reserve · in-place writes · adaptive notification — what each one fixes is the next slide
Kernel → userspace

BPF Ring Buffer — the new architecture that became the default (v5.8+)

An MPSC (Multi-Producer Single-Consumer) design where all CPUs share one buffer — answering Perf Buffer's four limitations with four mechanisms.

MPSC

One global shared buffer

Every CPU writes to one buffer; userspace reads it — global event ordering and memory efficiency at once.

ORDER + MEMORY
RESERVE / SUBMIT

Two-phase zero-copy

reserve() claims space first — if short, it fails immediately so the program can react. Write directly into buffer memory, then submit() — the intermediate copy is gone.

§11 REF TRACKING
ADAPTIVE

Adaptive notification

If the consumer is already processing, the signal is suppressed; wake only when it lags — notification cost scales with consumer state, not data volume.

NO SIGNAL STORM
DOUBLE MAP

Virtual-memory double mapping

The same physical pages are mapped twice, back to back — records straddling the boundary read contiguously, no wrap-around handling.

ZERO WRAP COST
Every reserve must end in submit/discard — enforced by the Verifier's reference tracking (§11) Reordering · over-allocation · late loss detection · wakeup storms — all resolved by the single-buffer design
Side by side

Ring buffer generations — same name, different design

Perf Buffer is a ring buffer inside, too — what the eBPF Ring Buffer changed is not the ring itself but everything about how rings are placed, written, and read.

Legacy ring — Perf BufferBPF Ring Buffer (v5.8+)
LayoutIndependent ring per CPU — as many rings as CPUsOne global ring — MPSC, shared by all CPUs
Write pathAssemble record on the stack, then copy into the bufferreserve() → write directly into the buffer → submit() — zero-copy
Out of spaceDiscovered only at copy time — assembly cost already paidreserve() fails immediately — the program can react
Event orderingNo cross-CPU guarantee — fork → exec can reverseSingle buffer — global order guaranteed
Boundary handlingRecords split at the boundary — read twice and reassembleVirtual-memory double mapping — contiguous read, no split
Consumer notificationUnconditional signal per eventAdaptive — wake only when the consumer lags
Consumer APIs look alike — perf_buffer__poll ↔ ring_buffer__poll; migration is cheap Ring Buffer is the default for new development — Perf Buffer only for pre-v5.8 kernel compatibility
Compile once — run everywhere

CO-RE — compile once, run everywhere

The root problem: kernel struct offsets differ across versions and configs. CO-RE recomputes the offsets at runtime.

COMPILE TIME · DEV MACHINE BPF C source includes vmlinux.h Clang -g emits relocation records .o (ELF) .BTF + .BTF.ext "which fields does it touch" RUNTIME · TARGET NODE The kernel's own BTF /sys/kernel/btf/vmlinux libbpf recompute offsets · patch instructions Kernel load offsets fit this kernel
BTF dedup in 5 stages — strings → non-reference types → struct/union canonicalization → reference types → compaction; hundreds of MB of DWARF down to a few MB Measured — x86_64 pt_regs offsets (di=112 · si=104 · dx=96) patched into the bytecode at load time Slim static binaries with no runtime compilation — unlike BCC (BPF Compiler Collection) embedding clang
The whole picture

The integrated flow — where the pieces interlock

Overlaying everything so far: the load path (compile → verify → JIT) and the runtime channels (Helpers · Maps · Ring Buffer) on one slide.

USER SPACE Clang / LLVM §10 · .o + .BTF.ext libbpf loader §20 · offset patching Kernel BTF /sys/kernel/btf/vmlinux the kernel's type dictionary Userspace consumer §18 · Ring Buffer reads · map polling KERNEL SPACE bpf() syscall BPF_PROG_LOAD Verifier §11 · exhaustive CFG walk reference tracking JIT §10 · native machine code BPF VM execution §13 · Helpers / KFuncs · Maps MPSC Ring Buffer §18 · reserve / submit · adaptive notification
Top row is the load path, bottom row the runtime channels — § numbers point to the detail slides Without patching the kernel — verified bytecode lives inside it
The frontier

Recent features pushing the boundary

Privilege delegation, large memory, userspace execution — eBPF's constraints are being lifted one by one.

KERNEL v6.9+

BPF Token

Instead of granting all of CAP_BPF, issue a delegation token — a container inside a user namespace loads only the allowed program and map types. Least privilege for multi-tenant environments.

DELEGATION
KERNEL v6.9+

BPF Arena

A large memory region where kernel and userspace share the same pages. Enables pointer-based data structures (trees, lists) instead of the map API.

SHARED MEMORY
EXPERIMENTAL

bpftime

Kernel uprobes trap via int3 with a heavy kernel round-trip — bpftime rewrites the binary to run probes in userspace, roughly 10× faster. Kernel entry is removed entirely.

USERSPACE RUNTIME
02
Chapter two

Observe and defend

landscape · memleak · off-cpu · throttling · falco vs tetragon · tuning

Tooling

The tool landscape — a ladder of abstraction

↑

bpftrace

DSL (Domain-Specific Language) one-liners — the standard for ad-hoc diagnosis. Use like awk, then throw away. bpftrace -e 'kprobe:do_sys_open {...}'

AD-HOC
·

BCC

Python + C framework with proven tools like memleak and offcputime. Caveat: runtime clang compilation — heavy to ship.

TOOLBOX
·

libbpf + CO-RE

The production standard — slim static binaries, no per-kernel recompilation. Cilium, Tetragon, and most commercial agents ship this way.

PRODUCTION
↓

bpftool

The standard CLI to inspect and dump programs, maps, and BTF loaded in the kernel. The answer to "what is attached to my kernel right now?"

INTROSPECTION
← quick experimentsproduction rollout →
Ecosystem

Product landscape — observe · detect · enforce

Products on the same eBPF stand in different places — how far they see, and where they start blocking, is the axis.

OBSERVE

Observability · profiling

Pixie — auto-instruments HTTP/gRPC/SQL with zero code change, builds service maps. Parca — always-on low-rate stack sampling, continuous profiling. Hubble — sidecar-free L3–L7 visibility (§36).

NO-INSTRUMENT
DETECT

Runtime detection

Falco — rule-based syscall behavior detection with the largest ruleset (§29). Tracee — Aqua Security; forensics-oriented detection with precise kernel event capture.

ALERT-ONLY
ENFORCE

In-kernel blocking

Tetragon — synchronous verdicts and blocking at LSM hooks (§30). KubeArmor — workload hardening that pre-empts file/process/socket access via BPF-LSM and AppArmor.

IN-KERNEL BLOCK
CORRELATE

Multi-dimensional — CADR

ARMO and peers — correlate kernel eBPF signals with cloud APIs, the K8s control plane, and app code; reconstruct attack stories instead of isolated alerts.

ATTACK STORY
The classification axis — does it ship events to userspace to 'report', or does it 'decide' inside the kernel? The structural gap between detect ↔ enforce is dissected in §29–30
Case study № 1

Tracking memory leaks — all aggregation stays in the kernel

How it works

  • Probes on the allocation paths — uprobes for user malloc·calloc·realloc·mmap, kprobes for kernel kmalloc → record address → (stack ID · size · timestamp) in a map
  • On free, delete the entry — long-lived entries = leak candidates
  • bpf_get_stackid() stores the call stack in a STACK_TRACE map
  • Userspace only reads the map periodically — no event streaming, minimal overhead
# track outstanding allocations of PID 1234 every 10 s $ memleak -p 1234 --interval 10 [13:42:01] Top 10 stacks with outstanding allocations: 524288 bytes in 128 allocations from stack malloc+0x1f grow_request_buffer+0x42 handle_connection+0x8d start_thread+0xd9 # --older: only "true leaks" unfreed for 60 s+ · omit -p to trace kernel leaks $ memleak -p 1234 --older 60000
Unlike Valgrind, no restart needed — attaches dynamically to a running process; in-kernel aggregation, no CPU emulation The cost of uprobes — an int3 breakpoint trap round-trips the kernel per call; at high malloc rates the trap itself is the bottleneck The workaround removes kernel entry — bpftime runs the probe as a userspace inline hook, ~10× faster (§22)
Case study № 2

Off-CPU analysis — seeing the time spent off the CPU

Task A timeline ON-CPU BLOCKED — locks · I/O waits the profiler's blind spot ON-CPU kprobe: finish_task_switch() the single point every context switch passes through record t₀ aggregate Δt Δt accumulated in a per-stack histogram map — tens of thousands of events per second digested in-kernel

Why it matters

  • On-CPU profiles see only half the story — waiting time never shows up in samples
  • Lock contention, disk I/O, page faults are often the real latency culprits
  • Record switch-out time → on return, aggregate Δt per stack, in-kernel
  • The deliverable is an Off-CPU Flame Graph — which stacks waited, and for how long
Case study № 3

Diagnosing CPU throttling — runnable but not allowed to run

CFS (Completely Fair Scheduler) Bandwidth Control — limits.cpu → quota / 100 ms period Running 40 ms quota spent THROTTLED — forced 60 ms wait runnable but unschedulable — whole cgroup Next period refill → resume runqlat — wait distribution from runnable to running normally tens of µs → skews right with 100 ms-class spikes under throttling throttled windows surface directly as the long tail of the run-queue latency histogram

Why the usual metrics miss it

  • The CPU utilization graph sits flat against the limit — busy and strangled look identical
  • cAdvisor throttle counters give counts and totals only — no idea which code path got delayed by how much
  • runqlat spikes + Off-CPU stacks (§27) showing no blocking traces (no io_schedule, no futex_wait) → not I/O, not locks: throttling, confirmed
  • The cross-reading of the two tools is your quantitative basis for resizing limits
Observed combinationDiagnosisPrescription
runqlat spikes + no blocking traces in stacks — awake but can't runCFS throttling — CPU allocation too smallRaising limits.cpu is effective
runqlat normal + stacks ending in io_schedule/futex_wait — went to sleep voluntarilyI/O / lock bottleneck — unrelated to CPURaising limits does nothing — fix storage or lock design
Runtime security № 1

Falco — sees everything, stops nothing

Process execve("/bin/sh") tracepoint sys_enter · argument snapshot Ring Buffer event stream Userspace rule engine verdict: "shell spawned in a container!" by the time the verdict arrives, the syscall has already finished executing time of argument check ≠ time of use → TOCTOU (Time-of-Check to Time-of-Use) race an asynchronous pipeline — detection only
Detection is strong — the largest ruleset and ecosystem But blocking is after the fact — by kill time, it's already too late
Runtime security № 2

Tetragon — decides in the kernel, blocks on the spot

Process execve("/bin/sh") LSM Hook called after args are final — no TOCTOU In-kernel Policy TracingPolicy CRD → BPF synchronous verdict Allow emit an event log only Block bpf_override_return → -EPERM bpf_send_signal → SIGKILL no userspace round-trip — verdict and enforcement complete in the same kernel context
Falco = after-the-fact detection Tetragon = synchronous execution blocking (enforcement) Kubernetes-aware — per-Pod/namespace policies via the TracingPolicy CRD (Custom Resource Definition) The next step is correlation — weaving kernel eBPF data with cloud API and app signals into full attack stories (ARMO CADR · Cloud Application Detection & Response)
Operations

Production checklist

01

JIT settings

net.core.bpf_jit_enable=1 — the default on most modern distros. bpf_jit_harden blinds constants against JIT spraying but costs performance — choose by trust boundary.

SYSCTL
02

Memory accounting — a kernel-version fork

Before v5.11: map memory counts against RLIMIT_MEMLOCK — raising ulimit -l is mandatory. v5.11+: switched to cgroup memory accounting — container memory limits now include BPF maps, so size them accordingly.

MEMLOCK
03

Shipping strategy

libbpf CO-RE static binaries instead of BCC runtime compilation. For older kernels without BTF, inject external BTF from BTFHub.

CO-RE
04

Overhead management

Filter as much as possible inside the kernel — cost jumps the moment events cross into userspace. Aggregate in per-CPU maps to eliminate lock contention.

OVERHEAD
03
Chapter three

eBPF on EKS

vpc cni · cilium · hubble · benchmarks

Already in production

The EKS eBPF landscape — already in production

eBPF is not something to adopt — it is already running in your cluster. Understanding it adds a debugging layer.

AWS · v1.14+

VPC CNI NetworkPolicy

The node agent attaches eBPF probes to each Pod's host veth. Policy enforcement with no iptables chains.

POLICY
CNCF

Cilium

The whole CNI (Container Network Interface) in eBPF — kube-proxy replacement, ENI (Elastic Network Interface) routing, O(1) lookups.

CNI
CILIUM

Hubble

L3/L4/L7 flow observability from the same eBPF datapath — service maps with zero extra instrumentation.

OBSERVABILITY
AWS MANAGED

Network Flow Monitor

An agent DaemonSet collecting TCP retransmits and RTT (Round-Trip Time) via the sock_ops hook.

MONITORING
eBPF on EKS № 1

VPC CNI NetworkPolicy — a division of labor: controller and agent

NetworkPolicy user-defined object Network Policy Controller EKS control plane · AWS-managed resolves selectors → concrete Pod IP sets PolicyEndpoints CRD — resolved results published WORKER NODE aws-network-policy-agent DaemonSet — watches PolicyEndpoints eBPF probes ingress / egress verdicts host veth Pod interface
No iptables chains — rule traversal cost does not grow linearly with policy count Troubleshooting step one: check the PolicyEndpoints CRD, not the NetworkPolicy Scope — Pod eth0 only · does not apply to host networking / Windows / Fargate
eBPF on EKS № 2

Network Flow Monitor — reading AWS's own eBPF in the source

The NFM agent is open source (Rust · Apache-2.0) — it shows exactly how chapter 1's concepts are used in production code.

HOOK

A single sock_ops program

BPF_PROG_TYPE_SOCK_OPS on cgroup v2 — no packet capture, only TCP-stack callbacks (connect · RTT · retransmit). Always returns BPF_OK — never touches customer connections (source comment)

§12 APPLIED
MAP

Map polling instead of a Ring Buffer

Per-socket stats accumulate in maps; userspace collects every 500 ms — not an event stream → the ring buffer isn't the answer

§18 COUNTERPOINT
SAMPLE

Sampling at the entrance only

Sampling is decided once, at new-socket entry — every event of a tracked socket is collected: internally consistent flow stats

CONSISTENCY
CAP

Drops its own privileges

After loading, drops CAP_SYS_ADMIN/CAP_NET_ADMIN, keeping only CAP_BPF for map reads — least privilege, executed

LEAST PRIV
License history rhymes — GPL-only-helper kernels (SUSE 15 SP5 · Ubuntu 20.04) reject the Apache-2.0 agent; the DTrace CDDL wall again (§6) Ops detail (NHI · enrichment · rollout) — Playbook NFM doc
eBPF on EKS № 3

Cilium ENI mode — an eBPF datapath with no overlay

Pod VPC-native IP eBPF (tc hook) service resolution · policy · O(1) lookup ENI → VPC routing no overlay encapsulation kube-proxy removed — iptables service chains gone Cilium Operator EC2 API — ENI/IP allocation CiliumNode CRD per-node IP pool Agent assigns Pod IPs

What changes

  • kube-proxy replacement — iptables O(n) chain traversal → eBPF hash-map O(1) lookup; lookup cost stays flat as services grow
  • ENI IPAM (IP Address Management) — Pods get VPC IPs directly, overlay overhead removed
  • Hubble — flow observability for free, from the same datapath
  • Absorbs mesh features — Gateway API · mTLS · Bandwidth Manager
Numbers, not vibes

eBPF in numbers — hands-on benchmarks

Measured by this playbook — EKS 1.31 · m6i.xlarge · 5 scenarios, median of 3 runs.

MetricA · VPC CNI (default)E · Cilium ENI + full tuningReading
TCP throughput12.41 Gbps12.40 GbpsEvery scenario saturates the NIC (12.5 Gbps) — throughput doesn't differentiate
Pod-to-Pod RTT p504,894 µs3,135 µseBPF datapath + tuning — ~36% lower
UDP loss (iperf3 saturation)20.4%0.03%Bandwidth Manager — EDT (Earliest Departure Time) rate limiting; a feature difference, not performance
HTTP p99 @1k QPS10.9 ms9.9 msThe gap narrows at the application level — the CNI is only part of the bottleneck
Conclusion: pick a CNI on latency · features · operating model — not throughput Service meshes follow the same arc — sidecars (a proxy per Pod) → eBPF + node proxies (Cilium Mesh · Istio Ambient)
Take these home

Read the kernel, and the cluster comes into focus

1

The Verifier is the contract

eBPF's safety is a load-time proof, not runtime isolation. Constraints like the 512 B stack and bounded loops are the price of that proof.

2

CO-RE changed shipping

BTF relocation ended per-kernel recompilation. libbpf static binaries are the production standard — keep BCC for experiments.

3

On EKS it's already table stakes

The NetworkPolicy agent, Cilium, and NFM all run on eBPF — and NFM's source is public. The moment you understand it, you gain a debugging layer.

One sentence through all three chapters — eBPF is a verified programming layer that changes the kernel's behavior without changing the kernel.

Wrap-up

The kernel is no longer a black box

Resources to go deeper — questions and discussion welcome.

Learn

eBebpf.io — What is eBPF?OVERVIEW
Kkernel.org BPF DocumentationKERNEL
BGBrendan Gregg — BPF Performance ToolsBOOK

Build on EKS

CCilium ENI ModeCNI
TTetragon — eBPF Security ObservabilitySECURITY
PBPlaybook — VPC CNI Deep DiveINTERNAL
THANK YOU Q & A Engineering Playbook — devfloor9.github.io/engineering-playbook