Why architecture matters here
The architecture matters because Kafka's durability and availability guarantees are only as good as the physical topology underneath them, and MSK's job is to give you a good topology by default. A single-broker or single-AZ Kafka loses data or availability when that node or zone fails. MSK spreads brokers across Availability Zones and replicates partitions across them, so a full zone outage — a real and periodic event — does not lose committed data and, with the right replication and acknowledgement settings, does not even interrupt the stream. Getting that topology right yourself is fiddly; MSK bakes it into the cluster shape.
It matters because the operational load of self-managed Kafka is exactly the kind of work that adds risk without adding product value. Broker replacement, rolling upgrades, security patching, and metadata-quorum management are necessary, error-prone, and utterly generic. Offloading them to AWS reduces the surface area where a human mistake — a botched upgrade, a mis-sized disk — takes down the streaming backbone that many services now depend on. The value of managed Kafka is measured in outages that didn't happen.
It matters because security and access control for a streaming platform are hard to retrofit and easy to get wrong. Vanilla Kafka's native auth (SASL, ACLs) is a separate world from your cloud IAM. MSK's IAM integration lets you express 'this service may produce to this topic' in the same policy language as 'this service may read this S3 bucket,' unifying identity and audit. That coherence matters when a platform carries sensitive event data across many teams.
It matters because storage economics dominate the cost of a busy Kafka cluster, and the architecture of retention decides the bill. Without tiered storage, keeping thirty days of a high-throughput topic means provisioning (and paying for, and managing) enough fast broker disk to hold all of it, most of which is cold. MSK tiered storage keeps only hot segments local and pushes cold ones to cheap S3, decoupling how long you retain data from how much expensive broker disk you buy — often a large cost reduction for replay-heavy workloads.
Finally it matters because the provisioned-versus-serverless choice is an architectural fork with real consequences. Provisioned MSK gives you control over broker type, count, and storage — and the responsibility to size them. Serverless MSK removes sizing entirely and scales with throughput, trading some control and per-unit cost for zero capacity planning. Choosing the wrong side means either paying for idle brokers or hitting limits you can't tune, so understanding the trade is part of the architecture.
The architecture: every piece explained
The broker cluster across Availability Zones. An MSK provisioned cluster is a set of Kafka broker nodes placed in the private subnets of your VPC, distributed across two or three AZs. Each broker hosts a share of the partitions. Clients — producers and consumers — reach the brokers over the network using standard Kafka bootstrap connections. Because the brokers live in your VPC, network access is governed by security groups and subnet routing, keeping the cluster private.
Partitions, leaders, and replication. Every topic is split into partitions, and every partition has a replication factor (typically three) meaning three copies live on three different brokers, ideally in three AZs. One replica is the leader that handles reads and writes; the others are followers that replicate the leader's log. The set of replicas that are caught up is the in-sync replica (ISR) set. Producers can require acknowledgement from all in-sync replicas (acks=all) so a write is only confirmed once it is durably on multiple brokers across zones — this is what survives a broker or AZ loss without data loss.
The metadata layer: ZooKeeper to KRaft. Kafka needs a consistent store of cluster metadata — which broker leads which partition, topic configs, controller election. Historically this was an external ZooKeeper ensemble that MSK ran for you. Newer Kafka (and newer MSK clusters) use KRaft, a built-in Raft-based metadata quorum that eliminates the separate ZooKeeper dependency, simplifies operations, and scales metadata better. MSK manages whichever the cluster uses, and the industry is migrating toward KRaft.
Tiered storage. MSK can keep recent log segments on fast broker-local storage (the hot tier) and automatically offload older, closed segments to S3 (the cold tier). Consumers reading recent data hit local disk at full speed; consumers replaying old data transparently read from the tier in S3. This decouples retention from broker disk capacity: you can retain data far longer without provisioning proportionally more expensive broker storage, and you can size brokers for throughput rather than for total retention.
Security and observability integrations. MSK encrypts data in transit (TLS between clients and brokers, and between brokers) and at rest (with KMS keys). For authentication it supports IAM (map AWS identities and policies directly to produce/consume permissions), SASL/SCRAM, and mutual TLS. It publishes broker, topic, and consumer-lag metrics to CloudWatch and can emit logs, so the same monitoring stack that watches the rest of your AWS estate watches Kafka. The MSK Connect service runs managed Kafka Connect connectors for moving data in and out without operating connector infrastructure yourself.
End-to-end flow
Follow an event from producer to consumer. An order-service instance wants to publish an 'order-placed' event. Its Kafka producer, authenticated to the cluster via IAM, resolves the leader broker for the target partition of the 'orders' topic (the partition chosen by the event's key, so all events for one order land in order). It sends the record over TLS to that leader.
The leader broker in AZ-1 appends the record to its local log segment and the two follower replicas in AZ-2 and AZ-3 pull the new record to replicate it. Because the producer requested acks=all, the leader waits until the in-sync replicas have the record before acknowledging. Only then does the producer's send complete successfully. At this point the event is durably stored on three brokers across three zones — losing any one broker or a whole AZ cannot lose it.
Time passes and the log segment holding this event ages past the hot-tier threshold. MSK tiered storage closes the segment and offloads it to S3, freeing the corresponding broker-local disk. The event is still fully readable; its location just moved from fast local storage to cheaper object storage, transparent to any consumer.
Meanwhile a fleet of consumers in a consumer group — say the fulfillment service — reads the 'orders' topic. Kafka assigns each partition to exactly one consumer in the group, so the partitions are divided among the instances. Each consumer reads records in offset order from its assigned partitions and periodically commits its offset back to Kafka, so if it restarts it resumes exactly where it left off. A consumer reading recent events hits the hot local tier; one replaying yesterday's events transparently reads the offloaded segments from S3.
Now inject a failure: the AZ-1 broker holding the partition leader dies. Kafka's controller (via KRaft or ZooKeeper) detects the failure and elects one of the in-sync followers in AZ-2 as the new leader. Producers and consumers, on their next metadata refresh, discover the new leader and continue against it. Because acks=all guaranteed the record was on the follower before it was acknowledged, no committed event is lost, and the stream continues with at most a brief blip while leadership moves — the whole point of the multi-AZ replicated topology.