Why architecture matters here
Ordering mistakes in Kafka are subtle and costly. "We use Kafka, so it's ordered" — until a repartition splits a user's events. "We use idempotent producer, so exactly-once" — until a consumer rewrites the same batch twice. Every guarantee has a scope; picking the right scope for your workload is design work.
The architecture matters because the pieces compose. Idempotence is per-producer; transactions extend it across partitions and topics; consumer read_committed isolation completes the EOS story. Miss one and the guarantee cracks.
With the pieces in your head you can design event pipelines that are correct as well as fast.
The architecture: every piece explained
The top strip is the ship path. Producer attaches keys to messages. Partitioner hashes the key to select a partition. Partition is the ordered append log; ordering is guaranteed only within a partition. Consumer group assigns partitions to members so each partition has one active reader — preserving order per key.
The middle row is the exactly-once machinery. Idempotent producer attaches a producer ID and sequence number so retries do not duplicate. Transactions group produces across topics and partitions into an atomic commit or abort. Isolation level read_committed skips uncommitted messages. EOS read-process-write combines input commit + output produce in the same transaction, giving true exactly-once for stream processing.
The lower rows are operations. Rebalance reassigns partitions when members join or leave; cooperative assignor avoids stop-the-world. Compaction / retention per topic decides log lifetime. Ops tracks consumer lag, rebalance frequency, and repartitioning plans for scale.
End-to-end flow
End-to-end: an event pipeline publishes user events to a topic with key = user_id and 24 partitions. Producer is idempotent. A stream processor reads events in transactions: reads a batch, computes aggregates, writes to two output topics, and commits the input offsets — all atomically. Downstream consumers use isolation.level=read_committed to skip uncommitted output. During a deploy, cooperative rebalance moves partition ownership without pausing others. Compaction keeps the latest per key on some topics for state recovery. Metrics show lag per partition per consumer.