Why architecture matters here
Pub/sub failures show up as consumer lag, ordering violations, or DLQ growth. Each is architecturally addressable: partition keys for order, capacity + consumer parallelism for lag, DLQ + retry policy for poison messages.
Governance matters at scale — schema registry, retention policies, and access control prevent the pipeline from becoming an untraceable swamp.
With the pieces mapped, pub/sub becomes a reliable substrate rather than a source of incidents.
The architecture: every piece explained
The top strip is the mechanics. Publisher produces events. Topic + partitions shards for parallelism; order within partition. Broker cluster holds durable storage. Consumers in groups get partitions assigned; offsets tracked.
The middle row is guarantees. Delivery guarantee — at-least-once, at-most-once, exactly-once — chosen per use. Ordering is per-partition; global order requires a single partition (limits throughput). Retention by size or time. DLQ catches messages that fail delivery.
The lower rows are governance. Schema registry versions events; compatibility rules prevent silent breaks. Metrics cover lag + throughput. Ops handles repartitioning, upgrades, and access.
End-to-end flow
End-to-end: order service publishes order-created events to topic orders with 24 partitions keyed by user_id. Fulfillment consumer group has 8 consumers; each takes 3 partitions. Analytics consumer group processes independently. Retention 7 days. Schema version 2; consumers on 1 or 2 compatible. Metrics show 12s lag on fulfillment, retention headroom, DLQ empty. On upgrade: consumers rolled sequentially; no downtime.