Why it matters

Kafka has become the standard event streaming platform. Understanding its model shapes many system designs.

Advertisement

The architecture

Topics: named event streams. Partitioned across brokers.

Partition: ordered, append-only log. Records identified by offset.

Broker: server hosting partitions. Cluster = many brokers.

Kafka fundamentalsProducerappends recordsPartition logordered, replicatedConsumerreads by offsetConsumers track their own offset; can replay by resetting
Kafka producer + broker + consumer.
Advertisement

How it works end to end

Replication: partition replicated to N brokers (leader + followers). ISR (in-sync replicas) determine availability.

Retention: time-based or size-based. Old records deleted; log compaction keeps latest per key.

Consumers: read at their pace, tracking offset. Can replay from any point.

Consumer groups: multiple consumers coordinate; each partition assigned to one consumer in group.