Kafka vs Pulsar vs Redpanda, in depth: storage models, what an acknowledged write means, scaling cost, consumption models and how to choose

Three storage models

Kafka stores each partition as a sequence of segment files on the local disk of the broker that leads it, and followers fetch from the leader to keep copies. The set of followers that are caught up is the in-sync replica set (ISR). Cluster metadata lives in a Raft-based controller quorum called KRaft; since Kafka 4.0, ZooKeeper mode has been removed entirely. Partition data and the broker that serves it are tied together, so the broker is both compute and storage.

Pulsar separates the two. Brokers are stateless: they own topics (grouped into bundles) for serving, but the data goes to Apache BookKeeper. Each topic is a chain of ledgers, and each ledger is written to an ensemble of E bookies, with every entry sent to Qw of them and acknowledged once Qa have persisted it. Metadata lives in a separate store: historically ZooKeeper, and now optionally Oxia, which the Pulsar project recommends for new clusters in the 5.0 line while ZooKeeper remains supported.

Redpanda implements the Kafka protocol in a single C++ binary built on a thread-per-core runtime. Each partition is its own Raft group, the controller is another Raft group inside the same process, and there is no JVM and no separate metadata service. Like Kafka, a node is both compute and storage for the partitions it hosts.

What an acknowledged write means

This is the question most comparisons skip, and it is where the systems genuinely differ by default.

In Kafka, a producer with acks=all is acknowledged when every replica in the current ISR has the record, and min.insync.replicas sets how small the ISR may shrink before writes are refused. Kafka does not fsync each write by default: the record is in the page cache of each replica, and durability comes from replication across failure domains. Losing every replica's page cache at once, for example a correlated power loss across racks, can lose acknowledged data. Most operators accept that because they spread replicas across zones.

Redpanda in production mode flushes to disk on a majority of the Raft group before acknowledging acks=all. Since version 24.1 it offers write caching, which acknowledges after replication but before the flush, explicitly trading that guarantee for latency; it is on by default only in development mode. So out of the box Redpanda's default is stricter than Kafka's, and with write caching it moves to roughly Kafka's model.

In Pulsar the bookie writes each entry to its journal and, with the default journal settings, syncs it before acknowledging, so an acknowledged entry is on disk on Qa bookies. The knobs that matter are E, Qw and Qa per namespace; a common production choice is E=3, Qw=3, Qa=2.

The practical rule: never compare latency numbers between these systems unless the durability settings are written next to them. A Kafka run without fsync and a Redpanda run with fsync are measuring different products.

Side by side

Scaling and rebalancing cost

Adding a node is cheap in all three; making it useful is not. In Kafka and Redpanda a new node receives traffic only when partitions move to it, and moving a partition means copying its data. A broker holding 6 TB of partition data, moved at a throttle of 100 MB/s so production traffic is not starved, takes about 17 hours. Tiered storage changes this sharply, because only the local hot portion moves; see Kafka tiered storage. Redpanda can balance partitions automatically; which balancing features are in which licence tier is a detail to check before you rely on it.

Pulsar's split pays off here. A new broker can take topic bundles immediately because ownership is metadata, and a new bookie starts receiving new ledgers immediately because each ledger picks its ensemble when it opens. The cost moves elsewhere: you now operate three services with different scaling curves, and a slow bookie or an overloaded metadata store degrades every topic that touches it.

Consumption models

Kafka's classic consumer group assigns each partition to exactly one consumer in the group. Ordering per key is simple, but parallelism is capped at the partition count, and one slow record blocks its partition. Rebalances pause consumption; the newer consumer protocol and cooperative assignment shrink that pause, as covered in consumer rebalancing. Kafka 4.2 made share groups (KIP-932) production-ready: many consumers read the same partitions, records are acknowledged individually, and delivery counts drive retries, which gives Kafka queue semantics. Before assuming Redpanda supports share groups, check the version you run.

Pulsar had queue semantics from the start. A subscription has a type: exclusive and failover give stream semantics, shared spreads messages round-robin with per-message acknowledgement, and key_shared spreads keys across consumers while keeping per-key order. Negative acknowledgements, redelivery delay and dead-letter topics are client features. If your workload is a work queue with slow, uneven jobs, Pulsar fits it without extra machinery.

# Kafka and Redpanda: the same Java producer config works against both
bootstrap.servers=broker-1:9092,broker-2:9092,broker-3:9092
acks=all
enable.idempotence=true
max.in.flight.requests.per.connection=5
linger.ms=5
compression.type=zstd

# Topic-level, set once per topic, with replication factor 3
min.insync.replicas=2

# Pulsar: a key_shared work queue with retries and a dead-letter topic (Python client)
import pulsar

client = pulsar.Client("pulsar://pulsar-proxy:6650")
consumer = client.subscribe(
    "persistent://payments/prod/settlements",
    subscription_name="settlement-workers",
    consumer_type=pulsar.ConsumerType.KeyShared,
    negative_ack_redelivery_delay_ms=30_000,
    dead_letter_policy=pulsar.ConsumerDeadLetterPolicy(max_redeliver_count=5),
)
while True:
    msg = consumer.receive()
    try:
        settle(msg.data())          # your idempotent handler
        consumer.acknowledge(msg)
    except TransientError:
        consumer.negative_acknowledge(msg)

Benchmarking honestly

Published benchmarks from vendors are tuned for the vendor. Run your own, with your message sizes, key distribution, retention and durability settings, on the instance types you will buy. The OpenMessaging Benchmark framework has drivers for all three and is the usual starting point. The method matters more than the tool:

1. Fix durability first: decide whether acknowledged means replicated or replicated and flushed, and configure every system to that same meaning.
2. Use realistic payloads and key skew. A uniform 1 KB benchmark hides the hot partition that will page you.
3. Measure tail latency (p99 and p99.9) at 50%, 75% and 90% of the throughput you need, not the peak throughput at which latency is already unbounded.
4. Include catch-up reads: run a consumer that starts 6 hours behind while producers continue. Historical reads competing with tail writes is where page-cache designs and tiered designs separate.
5. Kill a node during the run and record the latency spike and the time until replication is healthy again.
6. Run long enough for retention deletion, compaction and segment rolling to happen at least once.

Worked example: sizing one platform three ways

A company ingests 200 MB/s on average, keeps 3 days of history, runs about 40 teams on the platform, and needs a copy in a second region. Raw retained data is 200 MB/s x 86,400 s x 3 days = about 51.8 TB, and with replication factor 3 that is about 155 TB of disk if everything stays local.

With tiering keeping 6 hours local, local data falls to 200 MB/s x 21,600 s x 3 = about 13 TB, plus 51.8 TB once in object storage. This single decision matters more than the choice of system, and all three now support it, so plan with tiering whichever you pick.

On Kafka, the team would run three KRaft controllers and enough brokers for network headroom (a 200 MB/s ingest becomes 600 MB/s of replication and client traffic plus reads), use tiered storage, and use MirrorMaker 2 for the second region. Forty teams means investing in ACL and quota automation. On Pulsar, the 40 teams map naturally to tenants and namespaces with their own retention and quotas, and geo-replication is a namespace setting, but the team must run bookies and a metadata store well. On Redpanda, node count is likely lower for the same throughput and there is one binary to operate, but the team must confirm that the features it needs, such as tiering, balancing and cross-region replication, are in the licence tier it is paying for.

In this example the deciding factor was organisational: a platform team of three people with strong JVM and Kafka experience, so they chose Kafka with tiered storage. A team whose main need was per-tenant isolation and work queues would reasonably choose Pulsar.

Failure modes to plan for

• Kafka under-replicated partitions: a slow follower leaves the ISR, the ISR shrinks to min.insync.replicas, and the next failure makes producers fail with not-enough-replicas errors. Alert on ISR shrink, not only on offline partitions; ISR replication covers the mechanics.
• Kafka and Redpanda reassignment storms: moving many large partitions at once saturates disks and network. Throttle and move in waves.
• Pulsar bookie disk full: a bookie switches to read-only when its disks fill, ensembles route around it, and if enough do this the namespace cannot form an ensemble at all. Watch ledger disk usage and garbage collection lag.
• Pulsar metadata store overload: very large topic counts put load on the metadata store, which is part of why Oxia exists. Watch metadata latency as a first-class signal.
• Redpanda resource assumptions: a thread-per-core design expects dedicated cores and fast local disks; noisy neighbours and network-attached disks with poor fsync latency hurt it more than they hurt a page-cache design.
• Compatibility gaps: Kafka-protocol compatibility covers clients, but tooling that reads internal topics or uses newer protocol features may not behave the same. Test your Connect, Streams and admin tooling, not only producers and consumers.

Migration notes

Moving between Kafka and Redpanda is mostly a cutover problem, since clients do not change: mirror topics, move consumers with translated offsets, then move producers. Moving to or from Pulsar changes client code and semantics, so it is a migration project; Pulsar has offered a Kafka protocol handler, but treat it as a bridge to validate rather than a promise. In every case, make consumers idempotent first, as described in exactly-once semantics, because every cutover replays some records.

What to do next

1. Write down your durability requirement as a sentence about what may be lost in a correlated failure, and map it to each system's settings.
2. Estimate retained bytes with and without tiering; size local disk to the hot window only.
3. Decide whether you need queue semantics; if yes, compare Pulsar subscriptions with Kafka 4.2 share groups on your workload.
4. List your tenancy needs (teams, quotas, retention policies) and your cross-region requirement.
5. Run an OpenMessaging Benchmark test with matched durability, real payloads, a catch-up reader and a node kill.
6. Price the operational team as well as the hardware, and check licence terms for every feature you depend on.