Apache Kafka set the template, but Apache Pulsar and Redpanda answer real production pain points. Choosing between them depends on your operational model (managed vs self-hosted), your latency SLOs, and whether you need geo-replication out of the box.
Storage architecture
Kafka: log per partition, partition lives on one broker, replicated to N. Scaling = adding partitions. Pulsar: separates broker (stateless) from storage (BookKeeper). Brokers are interchangeable; you can rebalance in seconds. Redpanda: single-binary Kafka-compatible broker in C++, no JVM, no ZooKeeper. Inherits Kafka's storage model but with thread-per-core.
Latency profile
| Broker | p99 latency (1KB) | Why |
|---|---|---|
| Kafka | 5-15 ms | OS page cache + JVM GC pauses |
| Pulsar | 5-20 ms | Broker → BookKeeper hop |
| Redpanda | 1-5 ms | Thread-per-core, no GC |
Operational model
Kafka has the broadest tooling but is operationally complex — you tune JVM, ZooKeeper (until KRaft), and broker counts separately. Pulsar's broker/bookie split is conceptually cleaner but means more components. Redpanda is a single binary you ship — closest to 'just run it'.
Geo-replication
Pulsar wins here — multi-region replication is first-class. Kafka has MirrorMaker 2 (works, but extra cluster). Redpanda has cross-cluster replication via its own tooling. If your spec includes 'active-active across 3 regions,' start with Pulsar.
Decision matrix
Greenfield + low latency: Redpanda. Existing Kafka ecosystem (Kafka Streams, Connect, Schema Registry): stick with Kafka (Redpanda is wire-compatible too). Multi-region active-active: Pulsar. Want managed and not opinionated: Confluent Cloud or AWS MSK.