Google Cloud Storage internals, in depth: frontends, Spanner metadata, Colossus and how to design for them

The three layers

Frontends. Requests arrive through the Google Front End, which terminates TLS and routes to Cloud Storage API servers. These are stateless: they authenticate the caller, evaluate IAM and any VPC Service Controls perimeter, apply quotas and translate the JSON, XML or gRPC API into internal calls. Any frontend can serve any request, which is why they scale horizontally and why a retried request can land on a different server without harm.

Metadata. Every object is a metadata record: bucket, name, a generation number that changes whenever the object's data is replaced, a metageneration that changes whenever its metadata changes, size, CRC32C and MD5 checksums, storage class, custom metadata and a pointer to where the bytes live. Google moved this metadata from Megastore to Spanner, and has credited that move for strongly consistent object listing. A bucket's namespace is flat and sorted by name; slashes are just characters, which is why 'directories' are a listing convention, unless you create a bucket with hierarchical namespace enabled, which adds real folders and atomic folder rename.

Data. Object bytes are stored in Colossus. Google's public description of Colossus names curators that manage file metadata (stored in Bigtable), custodians that handle background durability work such as rebuilding lost chunks, and D file servers that hold the chunks on disk. Data is protected with erasure coding and spread across failure domains; Google has not published the exact coding parameters Cloud Storage uses, so treat any specific scheme you read about as unconfirmed. See Colossus in depth and Spanner.

The three layers

Frontends. Requests arrive through the Google Front End, which terminates TLS and routes to Cloud Storage API servers. These are stateless: they authenticate the caller, evaluate IAM and any VPC Service Controls perimeter, apply quotas and translate the JSON, XML or gRPC API into internal calls. Any frontend can serve any request, which is why they scale horizontally and why a retried request can land on a different server without harm.

Metadata. Every object is a metadata record: bucket, name, a generation number that changes whenever the object's data is replaced, a metageneration that changes whenever its metadata changes, size, CRC32C and MD5 checksums, storage class, custom metadata and a pointer to where the bytes live. Google moved this metadata from Megastore to Spanner, and has credited that move for strongly consistent object listing. A bucket's namespace is flat and sorted by name; slashes are just characters, which is why 'directories' are a listing convention, unless you create a bucket with hierarchical namespace enabled, which adds real folders and atomic folder rename.

Data. Object bytes are stored in Colossus. Google's public description of Colossus names curators that manage file metadata (stored in Bigtable), custodians that handle background durability work such as rebuilding lost chunks, and D file servers that hold the chunks on disk. Data is protected with erasure coding and spread across failure domains; Google has not published the exact coding parameters Cloud Storage uses, so treat any specific scheme you read about as unconfirmed. See Colossus in depth and Spanner.

Where strong consistency comes from

Because an upload writes all bytes durably first and then commits one metadata transaction that publishes the new generation, readers never see a partial object, and once the client receives success every read, metadata read, list and delete reflects it. The documented guarantee is strong global consistency for read-after-write, read-after-metadata-update, read-after-delete, and bucket and object listing.

This matters for design. Pipelines ported from object stores that were once eventually consistent often carry defensive machinery: list-after-write retry loops, manifest files that exist only to work around stale listings, and sleep calls before readers start. On Cloud Storage that machinery adds latency and code without adding safety. What you still need is a way to know a set of objects is complete, which is an application question, not a consistency one; a marker object written last answers it.

There is one well-known exception: caching. A publicly readable object served with a Cache-Control header that allows caching can be served stale by caches, including Cloud CDN, until the cache entry expires. If you overwrite public objects in place, set short cache lifetimes or, better, publish new content under new names and switch a small pointer object.

Generations, preconditions and safe concurrency

Objects are immutable; an overwrite creates a new generation. Generations turn every write into an optional compare-and-swap. The precondition ifGenerationMatch makes a write succeed only if the live generation is the one you read, and ifGenerationMatch=0 means 'only if the object does not exist'. A failed precondition returns HTTP 412 and changes nothing. This gives you optimistic concurrency without a lock service:

from google.cloud import storage
from google.api_core.exceptions import PreconditionFailed
import json

client = storage.Client()
blob = client.bucket("acme-config").blob("pipelines/state.json")

def update_state(mutate, attempts=5):
    for _ in range(attempts):
        blob.reload()                                   # fetch current generation
        gen = blob.generation
        state = json.loads(blob.download_as_bytes(if_generation_match=gen))
        mutate(state)
        try:
            blob.upload_from_string(json.dumps(state), if_generation_match=gen)
            return state
        except PreconditionFailed:
            continue                                    # someone else won; re-read and retry
    raise RuntimeError("too much contention on state.json")

# create-once semantics, safe to retry after a timeout:
client.bucket("acme-lake").blob("raw/7f3a/2026-10-05/batch-0007.parquet") \
      .upload_from_filename("batch-0007.parquet", if_generation_match=0)

Preconditions also make retries safe. Google's client libraries only retry non-idempotent operations automatically when a generation precondition is supplied, because otherwise a retried upload after a lost response could overwrite a newer write. With object versioning enabled, older generations are kept as noncurrent versions; with soft delete, which new buckets get with a seven-day default retention, deleted objects can be restored during the retention window. See object versioning.

Uploads and downloads at scale

Single-request uploads are fine for small objects. For anything large or sent over an unreliable network, use a resumable upload: the client opens a session, receives a session URI, and sends the data in chunks that must be multiples of 256 KiB (except the last). If the connection drops, the client asks the session how many bytes were persisted and continues from there. The object becomes visible only when the final chunk is committed.

To go faster than one stream, split a file into parts, upload the parts in parallel, and combine them with compose, which accepts up to 32 source objects per request and can be applied repeatedly. Composite objects carry a CRC32C but no MD5, so validate with CRC32C. gcloud storage cp performs parallel composite uploads for large files when configured to. The maximum object size is 5 TiB.

Downloads parallelise with ranged reads, which is what tools and connectors use to read big Parquet or model checkpoint files quickly. Always verify checksums end to end: request CRC32C validation in your client or compute it yourself, because a checksum computed only after the bytes landed protects nothing.

Request rates and key-range splitting

Cloud Storage spreads load by splitting the sorted key space into ranges served by different servers, and splits further as a range gets hot. A bucket starts with roughly 1,000 object writes and 5,000 object reads per second of capacity and autoscales beyond that; Google's guidance is to ramp gradually, not more than doubling the request rate every 20 minutes, so that splitting keeps up.

Because ranges are contiguous in name order, sequential names concentrate load. Object names that begin with a timestamp or an incrementing id send every new write to the end of the key space, which is one range. Put a short hash or a high-cardinality field first, for example raw/7f3a/2026-10-05T02:59:00Z.parquet rather than raw/2026-10-05T02:59:00Z.parquet. There is also a much smaller limit on repeated writes to the same object name; Google advises staying around one update per second per object, which rules out using one object as a high-frequency counter or lock.

Storage classes and locations

Storage classes are pricing and access contracts, not different durability: all classes are designed for eleven nines of annual durability. Location is the availability decision. Lifecycle rules, evaluated against metadata in the background, change class or delete objects by age, version count or prefix; lifecycle actions are asynchronous and can take time to apply. See lifecycle management and dual- and multi-region replication.

Worked example: a clickstream landing zone

A team ingests clickstream from 400 collectors into a data lake, about 30,000 files per minute, 2 to 20 MB each, read by Spark and BigQuery external tables. Their first design names files events/YYYY/MM/DD/HH/collector-N-seq.json and writes them all into one bucket on launch day.

That is 500 writes per second, within the starting capacity, but all names share a timestamp prefix, so writes pile onto the newest key range, and on launch day they will grow tenfold. The redesign does four things. Names start with a two-hex-character hash of the collector id, events/a7/2026-10-05/14/..., giving 256 independent ranges. Traffic is ramped over a few hours, doubling no faster than every 20 minutes. Every upload uses ifGenerationMatch=0, so collector retries never overwrite and never duplicate. A compaction job then rewrites each hour's small files into large Parquet files under a date-first prefix for readers, writing with preconditions and deleting sources only after verifying CRC32C. A lifecycle rule moves raw files to Coldline after 30 days and deletes them after 400.

Listing is prefix-based, so the hash costs readers a fan-out: to collect one hour the compaction job issues 256 parallel lists, events/00/2026-10-05/14/ through events/ff/2026-10-05/14/. Because listing is strongly consistent, those lists see every committed file, with no manifest needed. Each collector also writes one marker per prefix-hour, created with ifGenerationMatch=0, so the job knows every prefix is closed rather than merely non-empty.

Failure modes

• 429 and 503 errors during ramps. Retry with exponential backoff and jitter, and slow the ramp; persistent errors on one prefix mean the naming scheme is sequential.
• Lost update on shared objects. Two writers read-modify-write without preconditions; one update vanishes silently. Use ifGenerationMatch.
• Duplicate or overwritten data after retries. A timed-out upload is retried and replaces a newer generation. Use create-only preconditions.
• Stale public content. CDN or browser caches serve an old version after an overwrite.
• Surprise early-deletion charges. A lifecycle rule moves data to Archive and a cleanup job deletes it weeks later.
• Regional outage on a regional bucket. Data is safe but unavailable; only dual- or multi-region locations, or your own copy, keep it readable.
• Listing as a hot path. Listing a prefix with millions of objects is paginated and slow; services that list on every request should keep their own index in a database instead.

Trade-offs

The design gives you strong consistency and very high durability without any coordination code, at the price of per-object rate limits, request-based pricing and latency in the tens of milliseconds per operation. That makes it excellent for large immutable files and poor for millions of tiny objects updated constantly, which belong in a database such as Firestore, Bigtable or Spanner. Dual- and multi-region locations trade higher storage and replication cost for survival of a regional outage; hierarchical namespace buckets trade some flexibility for real folders and fast renames, which matters for Hadoop-style workloads. For mounting buckets as file systems, see Cloud Storage FUSE.

What to do next

1. Audit object naming for sequential prefixes on any path written faster than a few hundred objects per second.
2. Add ifGenerationMatch preconditions to every overwrite and ifGenerationMatch=0 to every create.
3. Switch large or long-distance uploads to resumable sessions and verify CRC32C end to end.
4. Plan request ramps for launches: start near 1,000 writes per second per bucket and double at most every 20 minutes.
5. Choose location by recovery objective, then class by access pattern, then add lifecycle rules.
6. Check soft delete and versioning settings so you know how deleted data can be restored, and what it costs.
7. Set Cache-Control deliberately on public objects and publish new content under new names.
8. Alert on 429 and 5xx rates per bucket and on lifecycle and replication lag where you depend on them.