Kafka in KRaft mode, three brokers per region, deployed with Ansible
Managed Kafka wanted more per month than my entire server bill. So AlertKick now runs its own: KRaft mode, three brokers per region, no ZooKeeper, deployed by the same Ansible that builds the rest of the fleet. Here is the reasoning, because “just use managed” and “just self-host” are both lazy answers.
Why Kafka at all
AlertKick’s API and its agent-facing endpoint services are separate processes on separate nodes. Commands flow one way (you toggle a security profile, an agent somewhere needs to hear about it), events and monitoring results flow the other. Both paths need a durable buffer in the middle: endpoints restart, nodes get redeployed, and a command must not evaporate because the consumer was mid-deploy when it was sent.
That is the actual requirement: a durable, replicated log between services, not web scale. Kafka is arguably heavier than the job needs, but it is the boring, known quantity in this space, the client libraries are mature, and one skill covers both the command path and the event firehose.
KRaft kills the ZooKeeper tax
Kafka’s historical price of admission was running a ZooKeeper ensemble next to it - a second distributed system to feed, monitor, and upgrade, purely as overhead. KRaft mode (Kafka’s built-in Raft consensus) removes it entirely. For a small cluster the brokers simply also act as controllers, combined mode, and the deployment is one systemd service per node instead of two clusters holding hands.
If you evaluated self-hosted Kafka years ago and walked away because of ZooKeeper, that objection is dead now.
The placement trick: ride the replica set
Each region already runs three database nodes for the MongoDB replica set. Three nodes is also exactly what a minimal HA Kafka cluster wants. So Kafka co-locates on the db nodes: replication factor 3, min in-sync replicas 2 for the topics that matter, and the cluster survives losing any single node without adding a single new server.
The Ansible mirrors the MongoDB structure the fleet already had: an install playbook, a configure playbook, a cluster-init playbook. The API’s broker list is templated per region from the fleet inventory, so a new region gets its brokers the same way it gets everything else.
The caveat is shared blast radius. A JVM and mongod on the same box will one day compete for memory, and the JVM’s default heap settings are written for machines far bigger than mine. Cap the heap explicitly, give the Kafka log dirs their own volume away from the Mongo data volume, and accept that this trade is right for the current scale, not forever. Budget hosting punishes you for ignoring memory twice as hard as real hardware does.
When managed Kafka is the right answer
To be fair to the option I rejected: if you have real throughput, a team whose time costs more than the subscription, or a compliance story that wants the broker infrastructure inside someone else’s audit boundary, pay for managed. The arithmetic flips fast once either of those is true.
But for a platform whose Kafka carries commands and telemetry rather than clickstreams, on infrastructure priced like Hetzner’s, the self-hosted version is three systemd services and some Ansible that took a weekend. The subscription would have been the single biggest line item in the budget, buying capacity I will not need for years.
Run the numbers for your own case rather than inheriting anyone’s dogma - but run them with KRaft in the picture, because half the internet’s “never self-host Kafka” advice was really “never self-host ZooKeeper”, and that cost is gone.