Kafka

Most Kafka sizing advice is hand-wavy. “Start with 3 brokers.” “Scale when you hit a bottleneck.” “Talk to your vendor.” I have spent years on both ends of that conversation, as the engineer asking and the person being asked, and the answer is almost always some version of “it depends, let’s just see how it goes.” That isn’t sizing. Its the easiest way to say “I don’t know.”.

“Which topic is costing us?” is the question every platform team gets, and nobody can answer it cleanly. SaaS providers bill you at the cluster level. Your finance team wants a line item per team. Your product managers want to know if their pipeline is the expensive one. You have a Kafka cluster, a bill, and no bridge between them.

In Hindu tradition, Chitragupta is the deity who maintains a complete record of every being’s actions. The divine accountant. Fitting name for a system that tracks exactly who used what and how much it cost.

Presented at Confluent Current 2025 in New Orleans. Cloud billing for Kafka is opaque by default. You get one invoice, shared across every team, with no clear way to tell who’s driving costs. This talk walks through how to go from that to transparent, identity-level cost attribution, covering the tooling, the data model, and the organizational patterns that make chargebacks and showbacks actually work.

Presented at Conf42 Observability 2025. Kafka doesn’t fail cleanly. It stalls, lags, and misfires beneath the surface. This talk cuts through the noise to show what signals actually matter, how to catch issues early, and how to make Kafka observable without drowning in metrics.

Featured on Confluent’s Life Is But A Stream podcast, February 2025. The batch vs. streaming question comes up constantly with teams evaluating Kafka. This episode breaks down the architectural trade-offs, when each approach makes sense, and what the transition from batch to real-time actually looks like in practice.

When multiple teams share a Confluent Cloud environment but a single CoE team manages everything, the billing question comes up fast: who’s actually using what, and how much does it cost?

Most monitoring setups for Kafka require long and lengthy parsing configurations to get JMX metrics into your telemetry system. This project standardizes that pipeline across Prometheus/Grafana, New Relic, Elastic/Kibana, Datadog, and OpenTelemetry.

Co-author and section reviewer. A comprehensive adoption guide for technical executives, platform owners, and project managers evaluating or rolling out Confluent. Covers architecture patterns, operational considerations, and the organizational side of moving to event-driven platforms.

Published on the Confluent Blog, March 2021. Self-managing a Kafka cluster means wiring up your own monitoring. This post walks through how to export JMX data from Confluent clusters into Prometheus and Grafana with minimal setup. It became the reference blog series for connecting Confluent ecosystems to Prometheus-based monitoring stacks.

Starting a Kafka cluster is easy. Sizing it correctly is where things get interesting. Most teams start with 3 brokers because that’s the minimum, and then hope it’s enough. This tool replaces the hope with math.