How to Pick a Message Broker, in depth: queue, log or pub/sub, requirement by requirement, with a worked decision

Three native shapes

Work queue. Messages are tasks. Each goes to exactly one of many competing workers, is acknowledged individually, and disappears once acknowledged. A message that is not acknowledged within a timeout is redelivered, possibly to another worker. Failed messages can be routed to a dead-letter queue. RabbitMQ classic and quorum queues and Amazon SQS are native work queues.

Log. Messages are facts appended to a partitioned, retained log. Consumers do not delete anything; each consumer group records an offset per partition. Any number of groups read the same data independently, a new group can start from the beginning, and a group can rewind to reprocess. Order is guaranteed within a partition. Kafka, Redpanda, Pulsar topics, Amazon Kinesis Data Streams, RabbitMQ streams and NATS JetStream streams are logs.

Pub/sub fan-out. Each published message is delivered to every current subscription. Delivery may be ephemeral (core NATS, Redis pub/sub: if nobody is listening, the message is gone) or durable per subscription (Google Cloud Pub/Sub, SNS feeding SQS queues, RabbitMQ exchanges bound to queues).

The shapes can imitate each other. A log with one consumer group behaves like a queue, except that acknowledgement is an offset, so one slow message blocks its partition. A queue per subscriber behind a fan-out exchange behaves like pub/sub. The question is which shape is native for your dominant workload, because imitations cost operational complexity.

The requirements that actually decide

Answer these for the main workload, not for every imaginable one. It is normal to end up with two systems, typically a log for business events and a managed queue for background jobs, and that is often cheaper than forcing one product into both roles.

How the common products map

Kafka and Redpanda are logs first. They shine at high-throughput event streams, many independent consumers and replay. Their weak spot is task-style processing: within a consumer group a partition is read by one consumer, so parallelism is capped by partition count, and one poison message stalls its partition until you skip or dead-letter it in application code. Kafka's KIP-932 adds share groups for queue-style consumption; check its status in the version you would run before relying on it.

RabbitMQ is a work-queue broker with flexible routing through exchanges, per-message acknowledgement, priorities and dead-lettering. Quorum queues replicate with Raft and are the recommended choice for durable queues. RabbitMQ streams add a log shape for replay, so one deployment can host both, though at lower scale than a dedicated log.

NATS has two layers. Core NATS is fast, ephemeral pub/sub and request-reply. JetStream adds persisted streams with consumers that can be push or pull, acknowledged per message, which covers queue and log patterns in one small binary. It suits edge, IoT and service meshes where operational simplicity matters.

Pulsar separates serving brokers from BookKeeper storage and supports exclusive, shared, failover and key-shared subscriptions, so the same topic can be consumed as a log or as a work queue. The price is more components to run.

Amazon SQS is a fully managed work queue: standard queues with at-least-once delivery and best-effort ordering, FIFO queues with ordering per message group and deduplication. Messages can be up to 1 MiB since an August 2025 increase. There is no replay; once deleted, a message is gone. Pair SNS or EventBridge with SQS for fan-out. Google Cloud Pub/Sub is managed durable pub/sub with per-subscription acknowledgement, ordering keys and a seek operation for limited replay.

Delivery semantics: plan for duplicates everywhere

Every broker in this list delivers at least once in its normal durable mode. A consumer can process a message and crash before acknowledging; the broker redelivers. Exactly-once features exist, such as Kafka transactions for read-process-write within Kafka, or FIFO deduplication windows, but they cover a narrow path and stop at the broker boundary; exactly-once semantics explains where they end. Any side effect in another system, a database row, an email, a payment, needs an idempotent consumer.

def handle(msg, db):
    # The message id must be stable across redeliveries: use a producer-assigned
    # event id, not a broker delivery id.
    event_id = msg.headers["event_id"]
    with db.transaction() as tx:
        inserted = tx.execute(
            "INSERT INTO processed_events (event_id) VALUES (%s) "
            "ON CONFLICT DO NOTHING", (event_id,)).rowcount
        if inserted == 0:
            return ACK                       # duplicate: already applied
        apply_business_change(tx, msg.body)  # same transaction as the marker
    return ACK                               # ack only after commit

The other half is the producer. Writing to the database and publishing to the broker are two systems, and a crash between them loses or duplicates the event. The outbox pattern writes the event to a table in the same transaction and relays it afterwards, which makes the broker choice independent of this problem.

Ordering: decide its scope before the product

Global order across all messages forces a single partition or queue and therefore a single consumer, which caps throughput. Almost no business needs it. What it needs is order per entity: all events for order 123 in sequence, while orders 123 and 456 proceed in parallel. Logs give you this with a partition key; SQS FIFO with a message group id; Pulsar with key-shared subscriptions; Pub/Sub with ordering keys. Kafka ordering shows how retries and producer settings can still break it.

Order also interacts with failure. If message 5 for an entity fails, processing 6 may be wrong. Ordered systems therefore block the key or the partition until the failure is resolved, and you need a policy: retry a bounded number of times, then park the message and every later message for that key, or skip it and alert. Write that policy down before choosing, because the products differ in how much of it they do for you.

Worked example: an online shop

A shop has two messaging needs. First, order events (created, paid, shipped, refunded) consumed by fulfilment, analytics, search indexing and a fraud model, with a new consumer added most quarters. Second, background jobs: sending emails, generating invoices, resizing images, each taking from milliseconds to a minute.

Size the order stream. Peak is 2,000 events per second at about 2 KB each, so 4 MB/s at peak. Sustained at peak that is about 346 GB per day; at an average of a quarter of peak it is about 86 GB per day. Seven days of retention at average load is roughly 600 GB, and three replicas make it about 1.8 TB of disk, well within a small log cluster or a managed log service. Analytics needs replay when a model changes, new consumers need history, and per-order order matters. That is a log, keyed by order id, with enough partitions for the largest consumer's parallelism, say 24.

Now the jobs. They are independent, of variable duration, need retries with back-off, a dead-letter destination for poison messages and occasionally a delay. Nobody needs to replay a sent email. That is a work queue; with the team already on AWS and no appetite for running RabbitMQ, SQS with a dead-letter queue is the low-effort answer. A consumer of the order log enqueues the jobs, so the two systems connect at one well-understood point. Dead-letter queues covers the redrive side.

Operational cost, honestly

Self-hosting a broker means owning upgrades, disk capacity, rebalancing, certificates, quotas, monitoring and recovery drills. A small team usually underestimates it. The minimum monitoring set is the same for every product: consumer lag or queue depth and its age, publish and consume error rates, redelivery or retry counts, dead-letter volume, and disk and memory headroom on brokers. Alert on the age of the oldest unprocessed message, not just on depth, because depth means nothing without throughput.

Managed services remove most of that list, at a price per request or per throughput unit and with less control over tuning and versions. A sensible rule is to start managed and self-host only when cost at your measured volume, or a missing feature, clearly justifies a team owning it.

Whichever you choose, rehearse failure before production does it for you. Kill a broker node under load and measure how long publishes stall; restart a consumer mid-batch and count duplicates; fill a dead-letter queue and run the redrive procedure end to end. These drills turn the semantics in the documentation into numbers your on-call engineers have actually seen, and they often reveal client settings, such as timeouts and retry limits, that matter more than the broker itself.

Failure modes

• Log as task queue. One slow or poison message blocks its partition while other partitions race ahead; consumers are capped by partition count.
• Queue as event store. A new service needs last month's events and they no longer exist anywhere.
• Assumed exactly-once. Duplicate emails or double charges after a consumer restart.
• Global ordering by accident. A single partition or FIFO group chosen for simplicity becomes the throughput ceiling.
• Unbounded retries. A poison message retried forever consumes capacity and hides itself in averages; always cap retries and dead-letter.
• Lag nobody watches. Retention expires on data a stuck consumer never read, and it is lost silently.

A decision procedure in code

The procedure is simple enough to write down, which makes the reasoning reviewable:

def pick_shape(r):
    if r.needs_replay or r.consumers_added_over_time or r.many_independent_readers:
        shape = "log"
    elif r.independent_tasks or r.needs_delay or r.needs_priority:
        shape = "work queue"
    elif r.latency_critical and r.loss_tolerable:
        shape = "ephemeral pub/sub"
    else:
        shape = "durable pub/sub"
    hosting = "managed" if not r.team_runs_stateful_clusters else "either"
    return shape, hosting, "key by " + r.ordering_entity if r.ordering_entity else "unordered"

What to do next

1. List your messaging workloads and, for each, answer the requirement questions above in writing.
2. Decide the native shape per workload; accept two systems if the answers diverge.
3. Name the ordering entity and the failure policy for a stuck key before choosing a product.
4. Size throughput, message size and retention with real numbers and compare against managed offerings first.
5. Make every consumer idempotent with a producer-assigned event id, and add an outbox on the producer side.
6. Set up lag-age, error, retry and dead-letter alerts before the first production message.