The bug where commands silently stopped reaching agents
Our brokers and topics were healthy. Our messages were gone.
Last month I fixed the most instructive bug of this project so far, and it is instructive precisely because every dashboard stayed green the whole time. If you run anything with a message queue in the middle, the failure shape is worth internalising.
The setup
When you change something in AlertKick that an agent needs to know about - push a config, trigger an upgrade, refresh a security ruleset - the command flows API -> Kafka -> endpoint service -> WebSocket -> agent. Every command type shares one publish helper, one topic, one consumer in the endpoint.
We had just finished migrating Kafka from per-node instances onto a proper regional cluster. The API’s producer was moved to the new regional brokers. The endpoint’s consumer config, templated separately, was not - it still pointed at the old per-node broker address.
Same topic name, different clusters, zero overlap.
The API published every command successfully, to a log nothing would ever read. The endpoint consumed diligently from a broker where nothing new arrived. Kafka on both sides did exactly what it was told, reported healthy, and the product feature quietly died.
Why nothing caught it
Every component check passed, because every component was fine:
- Producer metrics: publishing successfully. True.
- Consumer: connected, no lag on its broker. True - nothing to lag on.
- Broker health on both clusters: green. True.
The failure existed only in the relationship between components, and nothing was monitoring the relationship. There is no error to alert on when a producer and consumer disagree about which cluster to meet at. Each side’s contract was individually satisfied.
It got worse. Before the migration, the old setup had “mostly worked”: the API and endpoint shared a broker on the same node, and session stickiness usually landed both on the same box. When it occasionally didn’t, commands occasionally vanished. So there was a pre-existing background hum of “pushes are sometimes flaky”, and the migration turned sometimes into always without changing the symptom’s shape. A total failure disguised as a known intermittent one is remarkably good camouflage.
The fix, and the actual lesson
The code fix was one template: point the endpoint at the regional brokers, give each pod its own consumer group. Because every command type shares the same publish path, repairing it repaired all of them at once - config pushes, agent upgrades, rule refreshes, the lot. One chokepoint is a feature here: one fix, and no per-command wiring for anything added later.
The durable lesson is about monitoring. Component health tells you a machine is up. It cannot tell you your system works, because systems fail in the seams. The check that would have caught this in minutes is embarrassingly simple: periodically send a synthetic command end to end and alert if it fails to arrive. Publish a canary, consume the canary, measure the loop. That check monitors the path - producer config, cluster, consumer config, processing - as a single fact.
So now that check exists, and “does a message actually make it across?” joins my short list of things I verify after any messaging change, filed next to “did the backup actually restore”.
If your architecture diagram has an arrow between two boxes, ask what monitors the arrow.