Kafka

A common discussion As a Customer Success Technical Architect, I am often asked how to manage Kafka resources. The question usually comes up when a cluster is growing, a platform team is reviewing capacity, or an architecture needs to become more efficient: How can we reduce Kafka’s footprint without weakening the guarantees our applications need? Kafka can look deceptively light at the beginning. Create a topic, send a few records, and the cluster seems to have plenty of room. The picture changes as the platform grows. More topics bring more partitions. More replicas bring more copies of the data. More consumers can bring more fetch traffic and consumer-group coordination. ...

Kafka records have timestamps, and most of the time we barely think about them. That changes quickly when a producer sends a timestamp in nanoseconds instead of milliseconds. A record can suddenly look like it came from hundreds of years in the future. Kafka may accept it, and the original mistake can stay hidden until it affects retention, log segments, or downstream consumers. KIP-937 addresses this problem directly. It separates timestamps in the past from timestamps in the future, so operators can allow legitimate historical data without also accepting dates that are clearly implausible. ...

A common question when designing a Kafka cluster is: how many partitions can one broker support? There is no single answer for every cluster. Kafka topics use partitions so producers and consumers can work in parallel. But every partition also creates work and state on the brokers that host it. The practical limit depends on broker resources, replication factor, Kafka version, and operational needs. Treat partition counts as an operational capacity limit, not only as a logical design choice. ...