Capturing security events with eBPF: from fork-storm noise to a usable signal

eBPF is a firehose pointed at your face.

AlertKick’s agent uses eBPF for host security visibility - process executions, file access, network activity, delivered from the kernel with negligible overhead and no kernel module. Getting those events flowing took a few weeks. Turning them into something an operator can act on has taken most of a year, and that gap is the honest story of building on eBPF. Three chapters, in ascending order of subtlety.

Chapter one: the agent that DDoSed itself

Raw eBPF events are unfriendly: a process exec is a PID, a command, some ancestry. To mean anything on a container host, events need enrichment: which container did this happen in? Our first enrichment implementation resolved container names by shelling out to docker inspect per event.

Every container event spawns a subprocess. Spawning a subprocess is itself an exec event, which the agent captures, and enriches, by spawning another subprocess. On a busy Docker host the agent became its own event storm: fork rate twenty times baseline, the internal event channel overflowing, tens of thousands of dropped-event warnings per hour bloating the journal. The warnings were logged per drop, so the logging amplified the flood it was reporting.

The fix is the pattern for all agent enrichment, forever: never exec on the hot path. The agent now keeps a container inventory refreshed on a slow ticker - one process listing every thirty seconds instead of one per event - and enrichment is a map lookup. Drop warnings were replaced by an atomic counter reported once a minute: “dropped 871 in the last 60s” is one line and more informative than fifty thousand.

The general rule: your observer participates in the system it observes. Any per-event cost that can generate events is a feedback loop with your name on it.

Chapter two: healthchecks, the loudest legitimate thing on a server

With the agent no longer flooding itself, the dominant remaining noise was Docker healthchecks. Every container with a healthcheck execs something like sh -c "wget -q http://localhost/livez" every few seconds, forever, on every host. To a detection rule watching for network reconnaissance tools, that is wget fired hundreds of times per half hour - a feed rendered unreadable by infrastructure doing its job.

The naive fix - stop alerting on wget - is disarmament, since attackers use exactly these tools. The real fix required the event to carry enough truth to distinguish probe from probe-shaped attack, and that pulled a thread through the whole stack:

• Full command lines. The BPF probe originally captured only the first argument, so events read “sh executed wget” with no URL - literally not enough information to classify. Capturing full argv inside BPF means fighting the kernel verifier’s rules about what you can prove safe to read, and pages that are not faulted in at capture time, with a userspace fallback for the gaps.
• Knowing what the healthcheck should be. The agent reads each container’s configured healthcheck command from Docker’s own metadata - without exec, per chapter one - and stamps matching events as healthcheck executions. Matching a shell-wrapped, env-var-expanded, redirection-sprinkled command line against a config declaration is fiddlier than it sounds; environment expansion alone broke the first version, because the captured argv contains the expanded URL while the config contains $CADVISOR_HEALTHCHECK_URL.
• Exempting narrowly. Detection rules skip a stamped healthcheck only when its targets are all loopback. A “healthcheck” reaching out to an external host still fires. Unstamped events still fire. Fail open, always: unclassifiable noise is annoying, but a silenced attack is fatal.

One production host turned out to be running two Docker daemons, one with a non-default data root, so metadata lookups silently missed half the containers until the enricher learned to ask Docker where its data actually lives.

Chapter three: deciding what not to send

The last layer is admitting most captured events matter to no one. Every file read in /usr/lib, every SIGCHLD, every timer signal - the kernel’s background hum. The agent now scopes at the source: file events only for writes under watched directories or touches of genuinely sensitive files, signals only from a consequential set, all before events enter the pipeline.

We kept that scoping in userspace rather than pushing filters into the kernel probes, and I would defend that trade: userspace filters compose with server-side rules and cannot fail to load. A broken BPF filter does not degrade gracefully - it takes the whole sensor dark. In-kernel discarding would save some CPU; it is not worth the blast radius until profiling says otherwise.

The yardstick

“Usable signal” has a concrete definition: an operator opening a host’s event feed should see only things worth a decision. Every layer above serves that one sentence. If you are entering this space, budget accordingly: getting events flowing took weeks, and making them readable has taken most of a year.