Why architecture matters here
Pulsar architecture matters because the separation of compute and storage changes operational trade-offs. Rebalancing brokers is fast because they're stateless. Adding storage is independent. Geo-replication is built-in. Get these benefits without the constraints Kafka's design imposes.
Cost is comparable to Kafka. BookKeeper storage plus broker compute is roughly what Kafka would need; tiered storage saves on long retention.
Reliability is strong when configured well. BookKeeper's replicated logs give durability; Pulsar's brokers handle client failover.
The architecture: every piece explained
Walk the diagram top to bottom.
Producer. Publishes messages to a topic.
Broker (stateless). Receives messages, coordinates writes to BookKeeper, serves reads. Owns topic ownership assignments but stores no message data itself.
BookKeeper Bookies. Distributed log storage. Each ledger replicated across bookies. Truly append-only.
Topics + Subscriptions. Multi-tenant, partitioned, hierarchical (tenant/namespace/topic). Each subscription tracks its own offset.
Sub types. Exclusive (one consumer), shared (round-robin), failover (primary + backup), key-shared (same key to same consumer).
Geo-replication. Configure cluster peers; messages replicate asynchronously between clusters. Multi-active possible.
Tiered Storage. Offload older segments to S3; brokers transparently read from tier.
Functions. Lightweight stream compute — write a Python/Java function; runs alongside brokers; consumes topic + produces topic.
ZooKeeper metadata. Cluster metadata: brokers, bookies, topic ownership, config.
Kafka-on-Pulsar. Compatibility layer letting Kafka clients talk to Pulsar. Migration path.
End-to-end message flow
Trace a message. Producer publishes to topic "orders". Client SDK picks a broker (from ZK-discovered list); sends message.
Broker A owns this topic partition. Writes message to BookKeeper: append to current ledger. BookKeeper writes to 3 bookies (ensemble); returns when 2 ack (write quorum).
Broker acks producer.
Consumer with shared subscription "billing". Broker delivers messages round-robin to subscribers. Each acks; broker advances subscription cursor.
Consumer failover: one shared subscriber crashes; broker redistributes.
Broker A fails. ZooKeeper detects. Ownership of the partition moves to Broker B. B reads from BookKeeper (unchanged); producers reconnect. Downtime: seconds.
Retention: 7 days hot in BookKeeper. Older segments tiered to S3. Consumer reading old data hits S3 transparently.
Geo-replication: cluster in eu-west-1 receives; replicates async to us-east-1. Cluster failure in one region doesn't lose data.