Sizing a Kafka Cluster from First Principles
Most Kafka sizing advice is hand-wavy. “Start with 3 brokers.” “Scale when you hit a bottleneck.” “Talk to your vendor.” I have spent years on both ends of that conversation, as the engineer asking and the person being asked, and the answer is almost always some version of “it depends, let’s just see how it goes.” That isn’t sizing. Its the easiest way to say “I don’t know.”.