Why it matters

Consumer group behavior determines processing guarantees, scalability, and failover. Getting it right shapes streaming apps.

Advertisement

The architecture

Group ID: consumers with same group ID coordinate. Each partition to one consumer.

Assignment: coordinator (broker) assigns partitions on group changes.

Consumer group dynamicsConsumer groupshared workPartition assignment1 partition → 1 consumerOffset committrack progressAdding consumers past partition count wastes; excess consumers idle
Group + assignment + offsets.
Advertisement

How it works end to end

Rebalance: partitions redistributed when consumers join/leave. Pause + reassign.

Offset commit: manual (recommended) or auto. Manual gives at-least-once vs at-most-once control.

Static membership: consumers keep their IDs across restarts. Reduces rebalances.

Cooperative rebalancing: incremental instead of stop-the-world.