Why architecture matters here
The reason to care starts with cost and ends with operability. Local broker storage is provisioned block storage, replicated three ways, sized for peak retention — the most expensive way to hold bytes that are rarely read. Object storage is an order of magnitude cheaper per gigabyte, durable by design, and effectively unbounded. Moving cold segments there means you stop paying premium prices to keep month-old data on SSDs across every replica. For a topic where 99% of reads hit the last hour but you must retain ninety days for compliance or replay, the savings are not incremental; they change what retention policies are even affordable.
The operational win is just as large and often more valuable than the raw cost. In a classic Kafka cluster, the amount of local data per partition is what makes cluster changes slow and risky. Adding a broker to spread load triggers partition reassignment that copies entire logs across the network; recovering a dead broker means its replacement re-replicates everything it held; an under-replicated partition after a failure takes as long to heal as it takes to move its bytes. When the local tier is only the hot working set, all of these operations shrink proportionally. A partition that logically holds terabytes but locally holds gigabytes reassigns in a fraction of the time, so clusters become elastic in a way they simply were not before.
The trade-off is that you have introduced a second, slower storage medium into the durability and read path, and that has to be reasoned about explicitly. A consumer reading the tail is unaffected — its data is in page cache, microseconds away. But a consumer that seeks to an old offset now waits on an object-store GET, which is milliseconds to tens of milliseconds and occasionally much worse under throttling. The system must decide which segments live where, keep an accurate map from offsets to remote objects, and guarantee that a segment is safely in the remote tier before it is deleted locally. Get that boundary wrong and you either serve stale reads, lose data, or leak orphaned objects that cost money forever.
Understanding the mechanism reframes how you size and operate a cluster. The right questions become 'how many hours of local retention cover my real-time and lag-recovery working set?' and 'what remote-read latency can my backfill jobs tolerate?' rather than 'how many disks do I need for ninety days?' Those are capacity and SLA questions about two tiers, and answering them well is what turns tiered storage from a cost lever into a genuine architectural upgrade.
It also changes the blast radius of a bad retention decision. When all history is local, over-retaining is a slow-motion disk emergency that eventually pages someone; when history is remote, over-retaining is a line item you can see and cap with object lifecycle rules. That visibility is itself a feature — storage growth stops being a cliff the broker falls off and becomes a budget you manage, decoupled from the health of the compute layer that serves live traffic.
The architecture: every piece explained
Top row: how a byte moves from hot to cold. A producer appends to the partition's active segment, which lives on the broker's local disk and, crucially, in the OS page cache — the reason tail reads are so fast. Segments have a bounded size; when the active one fills, the broker performs a segment roll, closing and sealing it and opening a new active segment. Only sealed, immutable segments are eligible to move: the remote tier receives a copy of each closed segment, uploaded to object storage as one or more objects. From this point the segment's bytes exist in two places until local retention expires and the broker deletes its local copy, leaving only the remote one.
Middle row: the machinery that makes remote offsets addressable. The RemoteLogManager is the broker component that drives uploads — it watches for newly sealed segments, streams them to the object store, and records that they are safely remote before local deletion is allowed. It maintains a remote index: the mapping from log offsets (and timestamps) to the remote objects that contain them, plus the offset and time indexes needed to seek within those objects. This metadata is the linchpin — without an accurate offset→object map, a consumer asking for offset N in the cold range cannot be served. A tail consumer reads recent offsets straight from the page cache; a backfill consumer seeking old offsets is routed to the remote tier, and the broker fetches the relevant object, serves the requested range, and streams it on.
Bottom rows: the two retention horizons and the ops surface. Local retention is now a short window — typically hours — sized to cover live consumers plus a margin for lagging ones so they can still be served cheaply from disk. Remote retention is the long horizon — weeks or months — governing how far back history remains available in the object store before lifecycle rules expire it. The two are configured independently, which is the whole point: you decouple 'how much do I keep hot' from 'how much do I keep at all.' The ops strip names the durable concerns: upload lag (how far behind the RemoteLogManager runs, which bounds how much local data you must retain), remote-read latency (the object-store GET time that historical consumers feel), object lifecycle (expiring cold data and cleaning up orphans), and broker rebuild speed (now fast, because only the local tier must be re-replicated).
One design consequence deserves emphasis: replication still governs the local tier, but the remote tier's durability comes from the object store, not from Kafka's replicas. A sealed segment is typically uploaded once by the leader, and its durability is delegated to S3's own redundancy. This is why the boundary logic — never delete locally until the remote copy is confirmed, and never trust a remote read whose index entry is missing — is safety-critical rather than a nicety: the two tiers have different durability owners, and the handoff between them is where data can be lost if it is done carelessly.
End-to-end flow
Trace a topic partition through a full lifecycle. Producers stream events into the active segment; consumers tailing the topic read those same bytes from the page cache with sub-millisecond latency. Everything is local, exactly as in a classic cluster. The active segment grows until it hits its size or time bound, and the broker rolls it — the segment is sealed, immutable, and now a candidate for the remote tier.
The RemoteLogManager notices the newly sealed segment and begins uploading it to object storage, along with its offset and timestamp indexes. When the upload completes and is recorded in the remote-log metadata, that segment is officially in the remote tier. The broker's local-retention policy — say, six hours — continues to hold the segment locally until it ages out; only then is the local copy deleted. From that moment the partition's bytes for that offset range exist solely in the object store, but the log is logically unbroken: a consumer can still request any offset from the start of remote retention onward.
Now the two read paths diverge, and this is the heart of the design. A real-time consumer sits at the tail, reading offsets that are minutes old; those live in page cache and it never touches the remote tier — its latency and throughput are identical to a non-tiered cluster. A backfill job, however, seeks to an offset from three weeks ago to reprocess a month of data. The broker consults the remote index, finds the object containing that offset, issues a GET to the object store, receives the segment (or the relevant byte range), serves the requested records, and moves to the next object as the consumer advances. The consumer sees a normal fetch response; underneath, the broker is paging cold segments in from S3. Throughput here is bounded by object-store bandwidth and the broker's ability to prefetch, not by local disk — which is often faster for large sequential scans than a single local disk would be, because object stores parallelize well.
The performance character is worth stating precisely. Tail latency is unchanged because the hot path never left local memory. Historical-read latency has a floor set by the object-store round trip — the first byte of a cold seek costs a GET, so a consumer that constantly seeks to random old offsets pays that cost repeatedly, while one that scans sequentially amortizes it across large prefetched ranges. This is why tiered storage is excellent for backfills and replays (sequential, amortizable) and less ideal for workloads that do scattered point-lookups deep in history. The architecture optimizes for the common case — fast tail, cheap sequential history — and accepts a latency premium on the rare case of random cold access.
Consider the stress case that reveals the boundary logic. Suppose the object store experiences elevated latency or a partial outage while the RemoteLogManager is mid-upload and local retention is aggressive. If the broker were to delete a local segment before its remote copy was confirmed durable, that data would be gone — no local copy, no remote copy. The design forbids exactly this: local deletion is gated on confirmed remote presence, so under upload backpressure the local tier simply grows beyond its target rather than dropping unconfirmed data. The visible symptom is rising local disk usage and upload lag, not data loss — a self-throttling failure mode that is safe by construction, provided you monitor upload lag and provision enough local headroom to absorb an object-store hiccup.