Kafka

A common discussion As a Customer Success Technical Architect, I am often asked how to manage Kafka resources. The question usually appears when a cluster is growing, a platform team is reviewing its capacity, or an architecture needs to become more efficient: How can we reduce Kafka’s footprint without weakening the guarantees our applications depend on? Kafka can look deceptively light at the beginning. Create a topic, send a few records, and the cluster appears to have plenty of room. The picture changes as the platform grows. More topics bring more partitions, more replicas bring more copies of the data, and more consumers bring more network and coordination work. ...

Kafka records have timestamps, and most of the time we barely think about them. That changes quickly when a producer sends nanoseconds instead of milliseconds. Suddenly, a record can appear to come from hundreds of years in the future. Kafka may accept it, while the original mistake stays hidden until it starts affecting retention, log segments, or downstream consumers. KIP-937 addresses this problem directly. It separates timestamps in the past from timestamps in the future, so operators can allow legitimate historical data without also accepting dates that are clearly implausible. ...

A common question when designing a Kafka cluster is: how many partitions can a broker support? There is no single answer that applies to every cluster. Kafka topics are divided into partitions so that producers and consumers can work in parallel, but every partition also creates work and state for the brokers that host it. The practical limit depends on the broker resources, replication factor, Kafka version, and operational requirements, so partition counts should be treated as an operational capacity limit, not only as a logical design choice. ...