Why architecture matters here
The architecture matters because at exabyte scale, the metadata layer, not the disks, is the thing most likely to become the bottleneck. Any design that funnels all file operations through a single metadata master eventually runs out of memory and CPU on that one machine, capping the whole cluster. By sharding metadata across many curators backed by a scalable store, Colossus removes that ceiling: the file system can grow to hold vastly more files and chunks simply by adding metadata capacity, which is what lets a single Colossus cluster underpin products as large as BigQuery and Gmail simultaneously.
It matters because separating control from data is what keeps throughput unbounded. If the metadata service had to relay the actual bytes, its network and CPU would cap aggregate throughput regardless of how many disks existed. Because clients stream data directly to and from D-servers, total throughput scales with the number of storage daemons and disks, not with the metadata layer. The metadata service handles a comparatively tiny load of opens and lookups, so it can stay fast and scalable while petabytes move underneath it.
It matters because at this scale hardware failure is continuous, not exceptional. With tens of thousands of drives, some are always failing, and the system must treat that as normal operation rather than an incident. Colossus builds durability into the storage layer — replication or erasure coding of every chunk — and runs background custodians that constantly detect lost chunks and rebuild them. Durability and self-healing are properties of the architecture, so the loss of any individual machine or drive is invisible to the products built on top.
The architecture: every piece explained
Curators (the metadata layer). Curators are the control plane: a horizontally sharded service that holds the file namespace, the mapping from files to chunks, and the locations of those chunks. Rather than keeping all of this in one master's memory, Colossus stores metadata in a scalable key-value store (historically a BigTable-like system), so metadata partitions across many curators. Clients contact curators to create, open, and locate files, and the curators return the chunk locations the client will then read from directly.
D-servers (the data plane). D-servers are lightweight, largely stateless daemons that manage the actual storage media on a machine — flash and hard drives — and serve chunk reads and writes. They hold bytes, not knowledge of the namespace; a D-server does not know which file a chunk belongs to, only how to store and return the chunk it was told to. Because they are simple and stateless with respect to metadata, D-servers scale out trivially, and clients talk to them directly for all bulk I/O.
Custodians (the maintenance plane). Custodians are background processes that keep the system healthy without being in the read/write path. They detect chunks that have lost redundancy (because a disk or machine died) and rebuild them, rebalance data to even out load and capacity across the cluster, garbage-collect space from deleted files, and manage data movement between tiers. This continuous, autonomous maintenance is what turns constant hardware failure into a non-event.
The encoding and tiering layer. Colossus stores each chunk with a chosen redundancy scheme: full replication (multiple copies) for hot, latency-sensitive data, or Reed-Solomon erasure coding (data split into fragments plus parity) for colder data, which achieves the same durability at far lower storage overhead than replication. It also tiers data by temperature across flash and HDD, keeping hot data on fast media and migrating cold data to cheaper spinning disk, so cost and performance are balanced automatically.
End-to-end flow
To write a file, a client first contacts a curator to create it and obtain placement — which D-servers and media the new chunks should go to, and with what encoding. The client then streams the data directly to those D-servers, which write it to their disks or flash. As chunks are written, the encoding layer applies replication or erasure coding so the data is durable across independent failure domains. The curator records the chunk locations and metadata; the heavy byte stream never touches the curator itself.
To read a file, the client asks a curator where the file's chunks live and receives the set of D-servers holding them. It then reads the chunks directly from those D-servers, often in parallel across many drives to get high aggregate throughput. For erasure-coded data, the client (or a helper) reconstructs the original bytes from the available fragments and parity. Again, the curator's role ends after the lookup — the data path is client-to-D-server, keeping the metadata service lightly loaded.
In the background, custodians continuously scan the cluster's health. When a drive or machine fails and takes chunks with it, the affected chunks fall below their target redundancy; custodians detect this and rebuild the missing replicas or erasure fragments onto healthy D-servers, restoring durability. They also move data: rebalancing to relieve hotspots, migrating cold chunks from flash to HDD, and reclaiming space from deleted files. All of this happens without involving clients or interrupting live reads and writes.
Because the products above Colossus — BigQuery scanning columnar files, Spanner storing its logs, Cloud Storage serving objects — all go through the same client library and the same control/data-plane split, they share this durability, self-healing, and tiering for free. A single Colossus cluster can serve wildly different access patterns at once because the metadata layer scales horizontally and the data throughput scales with the D-server fleet, so no one workload's growth caps another's.
The tiering and encoding decisions are themselves part of the live flow rather than a one-time choice. When data is first written it is often stored with replication on flash so it is fast to read while it is hot; as custodians observe that a chunk has cooled — it has not been read for a while — they transparently re-encode it with Reed-Solomon and migrate it to cheaper HDD, reclaiming the flash for newer hot data and cutting the storage overhead of the cold chunk from several-times replication down to a small parity fraction. If that cold chunk suddenly becomes hot again, the system can promote it back. All of this movement happens underneath the file abstraction: the products above see the same file with the same durability and are unaware that its physical representation and media changed, which is precisely what lets one storage layer be both fast for hot workloads and cheap for the enormous cold tail at the same time.