Why architecture matters here
The architecture matters because durability is not a feature you can bolt on later — it is the reason object storage exists, and it is produced entirely by how data is spread and repaired. Storing three full copies of every object gives you durability but triples your cost; erasure coding gives you comparable or better durability at a fraction of the overhead by storing parity fragments instead of whole replicas. The choice between replication and erasure coding, and the specific k-and-m parameters, is the single decision that sets both your cost per byte and your tolerance to correlated failures. Get it wrong and you either bleed money on redundancy or lose data in the first correlated outage.
It matters because the separation of metadata from data is what lets the system scale at all. The metadata service is small per object (a key, a version, a list of fragment locations) but must be strongly consistent and is queried on every single request; the data plane is enormous but only needs to store and stream bytes. Coupling them — putting metadata on the same nodes as data, or making the data path consult a heavyweight database per byte — collapses the scaling story. The decoupling lets you grow petabytes of capacity by adding disks without touching the consistency-critical metadata tier, and scale metadata throughput independently of raw storage.
It matters because failure is the steady state, not the exception. At the scale where object storage is interesting, disks are failing, machines are rebooting, and racks are losing power constantly. The architecture has to treat a dead disk as a routine event that the background repair loop absorbs without human involvement, not as an incident. This is why the scrub-and-rebuild subsystem is not an afterthought but a first-class part of the design: durability is a race between entropy degrading fragments and repair restoring them, and the system must always win that race by a wide margin.
Finally, it matters because the flat, immutable, eventually-listable model shapes what applications can and cannot do. There is no rename, so 'moving' a prefix means copying and deleting; there is no append, so logs are written as many small objects; listing a huge bucket is a paginated scan, not an instant directory read. Understanding these constraints up front prevents the classic mistakes of treating an object store like a filesystem — building deep fake directory hierarchies, doing read-modify-write on large objects, or relying on strongly-consistent listing that the system may only offer with caveats.
The architecture: every piece explained
Top row: the request path and where a key becomes locations. A client PUT or GET names a bucket and an object key. The frontend / API gateway authenticates the request, enforces bucket policy, handles range requests and multipart uploads, and routes to the right internal services. The metadata service is the brain: it maps the (bucket, key, version) to the list of chunks and, for each chunk, the set of fragment locations. For a write, the placement / hashing layer decides which data nodes will hold the new fragments, deliberately spreading them so that no two fragments of the same chunk share a failure domain.
Middle row: the durability machinery on the write path. A large blob is first chunked into fixed-size parts so that storage, repair, and parallel transfer all operate on manageable units. Each chunk is erasure coded into k data fragments plus m parity fragments, such that any k of the k+m fragments can reconstruct the chunk — so you survive the loss of any m fragments. Those fragments are spread across zones (racks, power domains, availability zones) so a whole-zone loss takes out at most one fragment per chunk. The data nodes write each fragment to disk, compute and store a checksum, and acknowledge.
Bottom-left: the commit point that gives immutability its teeth. Only after enough fragments are durably acknowledged does the system commit the metadata, atomically making the new object (or new version) visible. Before that commit the object simply does not exist to readers; after it, the object exists completely. There is never a visible partial object, which is exactly what makes concurrent writers and reader isolation trivial — the metadata flip is the single atomic moment of creation.
Bottom-right and ops: the loop that defends durability over time. A background repair and scrub process continuously reads fragments, re-verifies their checksums to catch silent bit rot, and — whenever a disk or node loss drops a chunk below full redundancy — reconstructs the missing fragments from the surviving ones and re-places them. The ops strip names what to watch: durability (are any chunks below their fragment target?), rebuild queue depth (is repair keeping up with failures?), tail latency (is the data path healthy?), capacity against erasure overhead (how much raw space the parity costs), and hot-key skew (is one object or prefix overloading a handful of nodes?).
End-to-end flow
Trace a single large object from upload to a later read, then to a disk failure that repair silently absorbs.
The write: a client PUTs a 500 MB video. The frontend authenticates and, because the object is large, drives a multipart upload, splitting it into parts the client uploads in parallel. Each part is chunked, and each chunk is erasure coded into, say, k=8 data and m=4 parity fragments. The placement layer selects twelve data nodes spread across at least four independent failure domains so no zone holds more than three fragments of any chunk. The nodes write their fragments, checksum them, and acknowledge. Once every chunk has enough acknowledged fragments to guarantee durability, the metadata service commits the object atomically. Only now does a GET for that key succeed — before the commit, the key returned not-found.
The read: later, a GET arrives, possibly a range request for just the middle of the video. The frontend authenticates, the metadata service resolves the key to its chunk list and fragment locations, and the data plane reads k fragments per needed chunk — it does not need all k+m, only any k — and reconstructs the bytes, verifying checksums as it goes. If some fragments are on slow or busy nodes, the reader can fetch a few extra fragments in parallel and use whichever k return first, turning erasure coding into a latency hedge as well as a durability mechanism. The reconstructed bytes stream back to the client.
A disk dies: one of the twelve data nodes loses a disk, taking one fragment of many chunks with it. Nothing breaks for readers — any k of the remaining eleven fragments still reconstructs each chunk — but every affected chunk is now one failure closer to the edge. The repair subsystem notices the missing fragments (through the metadata's record of expected locations and through scrub), reads k surviving fragments for each degraded chunk, recomputes the lost fragment, and writes it to a fresh node in an appropriate failure domain. Redundancy is restored to full without any client ever seeing an error.
Silent corruption caught by scrub: separately, a fragment on a healthy disk quietly rots — a bit flips with no disk error reported. The periodic scrub reads the fragment, recomputes its checksum, finds a mismatch, discards the corrupt fragment, and rebuilds it from the others exactly as it would a lost one. This is the crucial defense that pure replication without checksums lacks: without scrubbing, silent corruption accumulates invisibly until enough fragments of one chunk are bad to make it unrecoverable, and the eleven-nines promise quietly becomes a lie.