eBPF observability architecture, in depth: hooks, maps, in-kernel aggregation, CO-RE and running agents in production

eBPF lets you load small, verified programs into the Linux kernel and run them when an event happens: a syscall, a kernel function, a function in a user binary, a timer tick, a packet. For observability this changes where instrumentation lives. Instead of every team adding a tracing library and redeploying, one privileged agent per node sees every process on it, including the ones nobody wrote instrumentation for.

This article explains the architecture from the event to the dashboard, shows a complete program that builds a syscall latency histogram inside the kernel, and covers what goes wrong in production. An earlier version of this page quoted a flat 1 to 3 percent CPU overhead. There is no such constant: cost depends on which hook fires, how often, and what the program does, so this version shows how to measure it instead. Profiling with eBPF has its own guide in continuous profiling architecture, and packet processing with XDP is covered in eBPF and XDP.

A loader verifies and attaches programs to kernel or user hooks; programs aggregate into maps or stream through a ring buffer, and the agent enriches and exports.

The five parts of every eBPF tool

Five parts make up every eBPF observability tool. A program is written in restricted C (or Rust, or bpftrace's own language) and compiled to eBPF bytecode. The verifier walks every path through that bytecode before load and rejects anything that could read invalid memory, loop without a bound or run too long; then the kernel JIT-compiles it to native instructions. A hook decides when the program runs. Maps are kernel-resident data structures shared with user space: hash tables, arrays, LRU hashes, per-CPU variants, and ring buffers. The agent in user space loads programs, reads maps, joins kernel identifiers such as PIDs and cgroup IDs to pods and services, and exports metrics or spans.

The verifier is the reason this is safe enough to run on production hosts and also the reason eBPF code looks constrained. Loops must have provable bounds (supported since kernel 5.3), every pointer from a map lookup must be null-checked before use, stack space is limited to 512 bytes, and you call kernel functionality through a fixed set of helpers. A program that passes cannot crash the kernel through those rules, although it can still cost a lot of CPU if attached to something hot.

The five parts of every eBPF tool

Choosing a hook

The choice of hook decides stability, overhead and what you can see.

Hook	Fires on	Stability	Notes
Tracepoint	predefined kernel events, e.g. syscalls, scheduler, block I/O, tcp	stable-ish ABI	first choice when one exists; `bpftrace -l` lists them
kprobe / kretprobe	entry or return of almost any kernel function	none	breaks when functions are renamed or inlined
fentry / fexit	same as kprobe, via BTF trampolines	none	lower overhead, typed arguments; kernel 5.5+ on x86
uprobe / uretprobe	a function or offset in a user binary	tied to the binary	traps into the kernel each hit; costlier per event
USDT	static probe points compiled into apps	stable	only where the app ships them
perf_event	timer or hardware counter overflow	stable	basis of sampling profilers

A practical order: tracepoints for anything the kernel already exposes, fentry where you need a kernel function and the kernel is new enough, and uprobes only when the information exists nowhere else, such as plaintext before TLS encryption in SSL_write. Uprobes on hot functions are the most common source of surprising overhead, because each hit is an exception into the kernel rather than a few instructions.

Streaming events versus aggregating in the kernel

Programs can get data to user space in two ways, and the choice dominates cost. Streaming writes one record per event into a ring buffer (BPF_MAP_TYPE_RINGBUF, kernel 5.8+, shared across CPUs and ordered) or the older per-CPU perf buffer. User space sees every event, which you need for traces or audit logs, but you pay per event for the copy, the wakeup and the user-space processing, and when the consumer falls behind, events are dropped.

Aggregating in the kernel updates a counter or histogram bucket in a map and lets user space read the map every few seconds. A syscall that happens a million times a second costs a map update each time but only a few hundred map entries cross into user space per scrape. This is how tools turn very hot events into Prometheus-style histograms cheaply. Per-CPU maps remove contention entirely, at the cost of summing across CPUs when reading.

Design for the question. A latency histogram, a rate, or top talkers by bytes should be aggregated in the kernel. A request-response pair you want to turn into a span has to be streamed, and so the tools that generate HTTP spans from the kernel, such as OpenTelemetry eBPF Instrumentation (the project Grafana donated from Beyla in 2025) and Pixie, filter early and stream only matched protocol events.

A complete program: syscall latency histograms

The program below records per-syscall latency for one process as a log2 histogram, entirely inside the kernel. It uses the stable raw_syscalls tracepoints, an LRU hash for start timestamps so missed exits cannot leak entries forever, and a read-only global set by the loader.

// syscall_lat.bpf.c: per-syscall latency histogram for one process
#include "vmlinux.h"
#include <bpf/bpf_helpers.h>

#define MAX_SLOTS 32
const volatile u32 target_tgid = 0;          /* set by the loader before load */

struct hist_key { u32 syscall_id; u32 slot; };

struct {
    __uint(type, BPF_MAP_TYPE_LRU_HASH);
    __uint(max_entries, 16384);
    __type(key, u32);                        /* thread id */
    __type(value, u64);                      /* entry time, ns */
} start SEC(".maps");

struct {
    __uint(type, BPF_MAP_TYPE_HASH);
    __uint(max_entries, 8192);
    __type(key, struct hist_key);
    __type(value, u64);
} hist SEC(".maps");

SEC("tracepoint/raw_syscalls/sys_enter")
int on_enter(struct trace_event_raw_sys_enter *ctx)
{
    u64 id = bpf_get_current_pid_tgid();
    if ((u32)(id >> 32) != target_tgid)
        return 0;
    u32 tid = (u32)id;
    u64 ts = bpf_ktime_get_ns();
    bpf_map_update_elem(&start, &tid, &ts, BPF_ANY);
    return 0;
}

SEC("tracepoint/raw_syscalls/sys_exit")
int on_exit(struct trace_event_raw_sys_exit *ctx)
{
    u64 id = bpf_get_current_pid_tgid();
    if ((u32)(id >> 32) != target_tgid)
        return 0;
    u32 tid = (u32)id;
    u64 *tsp = bpf_map_lookup_elem(&start, &tid);
    if (!tsp)
        return 0;
    u64 us = (bpf_ktime_get_ns() - *tsp) / 1000;
    bpf_map_delete_elem(&start, &tid);

    u32 slot = 0;
    while (us > 1 && slot < MAX_SLOTS - 1) {  /* bounded loop, kernel 5.3+ */
        us >>= 1;
        slot++;
    }
    struct hist_key key = { .syscall_id = (u32)ctx->id, .slot = slot };
    u64 one = 1, *cnt = bpf_map_lookup_elem(&hist, &key);
    if (cnt)
        __sync_fetch_and_add(cnt, 1);
    else
        bpf_map_update_elem(&hist, &key, &one, BPF_NOEXIST);
    return 0;
}

char LICENSE[] SEC("license") = "GPL";

Build it with CO-RE (compile once, run everywhere): generate vmlinux.h from the running kernel's BTF, compile to a BPF object, and generate a libbpf skeleton that the loader includes.

bpftool btf dump file /sys/kernel/btf/vmlinux format c > vmlinux.h
clang -g -O2 -target bpf -D__TARGET_ARCH_x86 -c syscall_lat.bpf.c -o syscall_lat.bpf.o
bpftool gen skeleton syscall_lat.bpf.o > syscall_lat.skel.h
# loader (C): skel = syscall_lat_bpf__open(); skel->rodata->target_tgid = pid;
#             syscall_lat_bpf__load(skel); syscall_lat_bpf__attach(skel);
#             then iterate the "hist" map every few seconds and export buckets.

Two honest caveats. Both tracepoints fire for every syscall on the host, so the filter runs millions of times a second on a busy machine even though only one process is recorded. And the first-insert path can lose a count when two CPUs race on the same new key, which is acceptable for a histogram but not for billing. For ad hoc questions, bpftrace expresses the same idea in one line:

bpftrace -e 'tracepoint:syscalls:sys_enter_fsync { @s[tid] = nsecs; }
  tracepoint:syscalls:sys_exit_fsync /@s[tid]/ {
    @us[comm] = hist((nsecs - @s[tid]) / 1000); delete(@s[tid]); }'

Compile once, run everywhere: BTF and kernel versions

Kernel structures change between versions, so a program compiled against one kernel's struct layout used to break on the next. BTF, compact type information the kernel exposes at /sys/kernel/btf/vmlinux when built with CONFIG_DEBUG_INFO_BTF, lets libbpf relocate field offsets at load time. That is what lets an agent ship one binary for a fleet. Check three things before choosing a design: that BTF is present on every kernel you run, the oldest kernel version you must support (ring buffers and the CAP_BPF capability arrived in 5.8, fentry in 5.5), and whether managed node images enable everything the agent needs. Older fleets usually fall back to perf buffers and kprobes.

Worked example: a latency regression after a node upgrade

A payments API's p99 latency rises from 40 ms to 300 ms after a node pool upgrade, but only on some pods. Application metrics show time spent inside a handler that writes an audit record and calls fsync. Nobody wants to redeploy with new instrumentation during the incident.

First, the kernel-level HTTP metrics from the node agent confirm the pattern without any code change: the slow pods all sit on nodes from the new pool. Second, on one such node, the fsync one-liner above runs for 60 seconds. Its histogram for the API process shows most calls in the 256 to 511 microsecond bucket, with a second cluster at 131 to 262 milliseconds. On an old-pool node the second cluster is absent. Third, the syscall histogram program, attached to the same process for ten minutes and exported as a metric, shows the cluster appears in bursts every few seconds. That pattern points away from the application and toward the storage layer under the new image, where the team finds a changed volume type with lower provisioned IOPS. They move the audit log to a buffered writer with periodic fsync and pin the volume type.

The point is the sequence: fleet-wide signals from an always-on agent narrow the search, ad hoc bpftrace answers a precise question on one node, and a purpose-built in-kernel histogram turns that answer into a metric you can alert on. The resulting signals flow into the same pipeline as everything else, typically through an OpenTelemetry Collector.

Running agents in production

Run node agents as a privileged DaemonSet or, better, with only the capabilities they need: CAP_BPF and CAP_PERFMON on 5.8+ kernels, plus CAP_NET_ADMIN for network hooks; older kernels need CAP_SYS_ADMIN. Kernels before 5.11 also charge map memory against the locked-memory limit, so agents raise RLIMIT_MEMLOCK; newer kernels account it to the cgroup instead. Treat the agent as security-sensitive software: it can read any process's memory through uprobes.

Measure overhead instead of trusting a vendor number. The kernel can count time spent in each program:

sysctl -w kernel.bpf_stats_enabled=1     # small cost of its own; turn off afterwards
bpftool prog show                        # each program lists run_time_ns and run_cnt
# average cost per invocation = run_time_ns / run_cnt
# total CPU share = delta(run_time_ns) / (interval_ns * cpu_count)
sysctl -w kernel.bpf_stats_enabled=0

Set an explicit budget, for example a fraction of a percent of node CPU for the always-on agent, and alert when it is exceeded. Also export the agent's own health: ring buffer drop counts, map fill levels against max_entries, and attach failures per probe after every kernel upgrade.

Failure modes

Silent attach failure. A kprobe target was renamed or inlined in a new kernel, the probe never fires, and dashboards show zero instead of an error. Alert on probes attached versus expected.
Uretprobes on Go. Go's growable goroutine stacks conflict with uretprobe's return-address rewriting and can crash the process; Go-aware tools attach uprobes at each return instruction instead.
TLS blind spots. Uprobes on SSL_write see nothing in binaries that statically link their TLS stack or use a library the tool does not know, so HTTPS traffic from those services goes missing.
Full maps. A hash map at max_entries rejects new keys, silently dropping data. Use LRU maps for per-thread state and bound key cardinality for histograms.
Ring buffer drops. Bursts exceed what the consumer reads; counts derived from streamed events undercount exactly when load is highest.
Namespace confusion. bpf_get_current_pid_tgid returns host PIDs, not container PIDs; joining them to pods requires cgroup IDs or the agent's own mapping.
Missing symbols. Stripped binaries leave uprobes nothing to attach to and profiles unsymbolized.

Trade-offs

eBPF gives coverage without code changes, consistent kernel-level truth (retransmits, run-queue delay, disk latency) and one agent per node instead of one library per language. SDK instrumentation gives business context eBPF cannot infer: tenant, order ID, feature flag, and spans across asynchronous boundaries such as queues. Kernel-derived HTTP spans also struggle to propagate trace context through every runtime. Most mature setups use both: eBPF for the floor of coverage and infrastructure signals, SDKs where the business logic lives, joined by the trace model in distributed tracing architecture.

What to do next

Inventory kernel versions and BTF availability across every node pool; record the oldest you must support.
Try bpftrace on a staging node: list tracepoints, histogram one syscall for one process.
Pick one always-on agent for RED metrics, such as OpenTelemetry eBPF Instrumentation, and run it in one cluster.
Enable kernel.bpf_stats_enabled briefly, measure per-program cost, and write down a CPU budget.
Export agent health: probes attached, ring buffer drops, map fill, and alert on each.
Decide which services still need SDK instrumentation for business context, and document the split.

Key takeaway: eBPF observability runs verified programs at kernel and user hooks and moves their data out through maps or ring buffers to a node agent. Prefer tracepoints and fentry over kprobes and uprobes, aggregate in the kernel unless you need every event, ship one CO-RE binary that relies on BTF, measure per-program cost with kernel.bpf_stats_enabled rather than trusting a quoted percentage, and alert on the agent's own attach failures, drops and full maps.