An AI-built agent wedged a 2GB prod host: unbounded maps, OOM loops, and the fixes

Nothing humbles you like your own security agent OOM-killing a customer-visible host.

Last week a small production node (2GB of RAM, running a full application stack plus our own monitoring agent) ate itself in eight minutes. Memory climbed from 0.6GB to 1.9GB, the kernel started thrashing swap, load average hit 22, and the box wedged solid. The agent kept heartbeating just enough to report “online” while ignoring every command sent to rescue it. Recovery required a reboot.

The post-mortem found two bugs that I would bet money exist in code you are running too, plus an uncomfortable observation about AI-written code. All fixes shipped; here is the autopsy.

Bug one: maps that only grow

The memory climb came from in-memory maps keyed by things the outside world controls. The agent’s SSH monitoring tracked failed logins in maps keyed by source IP and username. Entries were pruned, but only the values; the keys were never evicted. An SSH brute-force flood, the most routine hostile traffic on the internet, minted a permanent map entry per attacking IP. The event deduplication cache had the same disease: a window with no size cap, drained on a timer, accumulating full event structures between drains whenever event cardinality spiked.

Each of these is attacker-influencable memory growth. The busier the attack, the faster the security agent, the thing meant to protect the host, consumed it. Nobody designed that; it emerged from “keep a map of what we’ve seen” written without asking who controls the keys.

Every fix is a bound: eviction on the key sets, a hard cap on the dedup window, a garbage-collection pass on the trackers. Boring. The lesson is the question that was never asked at review time: who controls this map’s keys, and what is its maximum size? If the answers are “the internet” and “unbounded”, you have shipped a remote memory-exhaustion primitive and called it a feature.

Bug two: the heartbeat that lied

The wedge had a crueller layer. The host looked online, heartbeats flowing, but ignored the unlock commands that would have let me in to fix it. Heartbeat responses and inbound commands travelled different paths through the agent. Heartbeats returned via a lightweight path that stayed alive under pressure. Commands went through a buffered channel into a single dispatcher goroutine that ran some handlers inline, including disk writes and multi-second collection calls. Under memory thrash, that dispatcher starved, the buffer filled, and new commands were silently dropped. The code even logged “dropping request” - into journals on a host too wedged to ship logs.

The tool for fixing the agent was the agent. Fixes: commands dispatch asynchronously, blocking work moved off the dispatcher, and drops made loudly visible. The deeper redesign lesson: liveness and readiness are different facts. A heartbeat that does not prove the command path works is a green light wired to nothing, and any protocol where “I’m here” and “I can act” can diverge will eventually report the first while failing the second, during exactly the incident where the difference matters.

The AI part, honestly

This agent’s code is substantially AI-written, reviewed and shipped by me. Both bugs are classes that code-generation models produce with total confidence and reviewers skim past: the map-with-no-eviction looks complete (it even prunes! just not the keys), and the channel-with-a-buffer looks defensive right up until you ask what happens when the buffer fills.

I have stopped reviewing AI code for whether it looks right - it always looks right, that is the failure mode. The review questions that would have caught both bugs are “what bounds this?” and “how will I know when it stops working?”, asked mechanically of every map, channel, and cache. AI agents multiplied my output enough to build this product solo; these two bugs are the tax notice. The tax is payable in review discipline, and it is still a very good deal.

Aftermath

Beyond the code: the host was undersized for its stack and is being resized - bounded code on a starved box is still a starved box. The dashboard gained an attention indicator so a host with open issues shows a warning even while “online”. And the agent now watches and reports its own resource footprint.

Audit your maps.