Paired pollers so a check is never missed - and the bug that taught the opposite lesson
In monitoring, a missed check is a lie you tell your customer. If their site went down during the gap, your product said everything was fine. So AlertKick’s uptime checking was built around one invariant: a check must never be silently skipped.
This month I got a masterclass in the invariant’s evil twin. A check that runs three times is also a lie, just a louder, flappier one.
The design: paired pollers
Uptime and response-time checks run from dedicated poller processes. Each region runs a pair: a leader executing checks, and a hot standby consuming nothing but ready to take over. Assignments arrive per region on the same message infrastructure as everything else, keyed per monitor so a given monitor’s checks stay sticky to a location. A poller dying mid-cycle loses nothing; work is queued, not held in the process.
The trade you accept with this shape is at-least-once delivery. Design for “never zero” and you must simultaneously design for “occasionally more than one”, which means dedupe and idempotency on the result path are load-bearing from day one. I knew that in theory. The bug found the gap between theory and practice.
The bug: the old scheduler never died
Before dedicated pollers existed, checks ran inside the API process itself - a legacy in-process checker. When the poller architecture shipped, the legacy path was kept behind a config flag, default off. Belt and braces.
Except the flag was never emitted into the production config template. Unset meant the legacy default won on every backend node. So on every one of them the old checker cheerfully kept running alongside the new poller fleet.
Every monitor was being checked roughly three times: once by its assigned poller, plus once per backend node by the resurrected legacy checker - each with its own node-local queue, invisible to the others.
What triple execution looks like from outside
Three racing writers on one monitor document produce distinctive garbage:
- Status flapping. Checkers disagree for a moment - one sees a timeout, another a success - and the monitor whipsaws ok, warning, critical, ok within seconds.
- Duplicate alert events seconds apart, offset by the skew between nodes.
- Corrupted stats. Total and failed counts inflated roughly threefold; a flaky-but-mostly-fine site accumulates a health score suggesting disaster.
- Poisoned location metrics. The legacy path reported under the monitor’s configured location label while running from EU nodes - so “US East” latency numbers were being measured from Helsinki.
I found it chasing a single “flaky” monitor. The tell was in the alert wording: the two code paths phrased their failure messages slightly differently, and both phrasings appeared in one monitor’s history. Two voices meant two schedulers. From there, log queries matched the legacy checker’s distinctive lines to the duplicate events to the second.
The fix, in three layers
The code now actively refuses to run the legacy scheduler when poller mode is on, and on startup deletes the legacy queue so stale work cannot linger. The config template now emits the flag explicitly on every environment, no silent defaults. And the one environment that genuinely has no poller fleet pins the flag the other way, on purpose, in versioned config - visible, not vestigial.
Verified the boring way: startup logs confirming the checker is disabled on every node, queue keys gone, the legacy path’s log lines flatlining at a known minute.
The lesson
When you replace a scheduler, decommissioning the old one is the migration. A replaced component that can still run is a fork of your write path waiting for a config default to set it loose. If you must keep an old path, make it require explicit opt-in, and make the new path actively assert the old one is off.
And “never miss a check” and “never duplicate a check” are two requirements in permanent tension. At-least-once delivery plus idempotent processing is the only honest resolution, and both halves have to ship.