Spanner architecture — TrueTime, Paxos groups, directories, external consistency

Why architecture matters here

Spanner surprises come from mismatching workload to architecture. Cross-region strong reads pay TrueTime waits. Cross-directory transactions pay 2PC. Hot rows cause split hotspots. The architecture matters because schema + region choices shape SLO.

With the pieces mapped, you can pick region config, use interleaved tables to avoid cross-directory 2PC, and choose read variants that match consistency needs.

The architecture: every piece explained

The top strip is the physical stack. Client SQL arrives via gRPC. Zone / region hierarchy holds replicas. Paxos group replicates each split (a range of a table). Directories are logical placement units that migrate with load.

The middle row is the correctness machinery. TrueTime API returns TT.now() = [earliest, latest] with bounded uncertainty. Commit wait for external consistency waits until latest < now on every node. 2PC across groups handles cross-split transactions. Read variants: strong (linearizable, expensive), bounded staleness (cheap, allows lag), stale (fastest, returns timestamp).

The lower rows are ops. Split management rebalances loaded splits. Observability exposes latency + hotspot metrics. Ops covers schema design (interleaved tables, primary key), and cost.

End-to-end flow

End-to-end: a client inserts a row. Spanner assigns to a split's Paxos group; leader commits via Paxos; commit-wait ensures external consistency. Cross-region single-directory write: ~50ms latency in a regional config. Cross-directory write: 2PC across groups, higher latency. Read with staleness = 5s: no TrueTime wait, fast. Strong read across regions: pays TrueTime wait + Paxos round.

What TrueTime gives you, and where it is derived

Everything below rests on one primitive: Spanner hands out commit timestamps that are globally comparable. If transaction A finished before transaction B started in wall-clock reality, A's timestamp is smaller -- on any machine, in any region, with no coordination between the readers. That is the whole contract this article needs.

The derivation -- the uncertainty interval, why commit wait is exactly the right amount of waiting, the clock sources that keep the bound small, and how snapshot reads exploit the resulting version history -- is worked through in Spanner TrueTime: global consistency through bounded clock uncertainty. Read that one for the clock; read this one for the database around it.

Splits: how a table becomes a distributed data structure

A Spanner table is a single sorted map from primary key to row, and that ordering is the partitioning function. Spanner cuts the sorted keyspace into contiguous ranges called splits, each an independent unit of replication, leadership and load. There is no hash bucketing, no shard count you pick, no resharding job you run: the system watches per-split size and load, cuts a hot or large split in two, and merges cold neighbours back. (The original paper calls the placement unit a directory; Cloud Spanner's tooling says split. Same idea -- a relocatable contiguous key range -- and the diagram above labels both.)

Two properties follow, and they shape every schema decision later. Adjacency is physical: neighbouring keys share a split, so a range scan over a key prefix is one sequential read rather than a scatter-gather. And a transaction inside one split is fundamentally cheaper than one that is not, because one split is one Paxos group and needs no cross-group agreement. Nearly all Spanner performance work reduces to making the second property hold more often.

Splitting is also reactive, with a real time constant. It observes traffic and then acts; it does not anticipate. A launch that goes from zero to peak in thirty seconds concentrates on a handful of splits for the first several minutes, which is why ramping traffic -- or warming the keyspace with representative load before cutover -- is the standard mitigation for a big-bang migration.

Replicas: read-write, read-only, and witness

Read-only replicas carry a full copy and answer snapshot reads locally, but take no part in the write quorum -- so adding them scales reads in a distant region without adding a wide-area hop to every commit. Witness replicas are the interesting one: they store the write log and metadata but not the row data. A witness lets a configuration reach majority across three failure domains while paying to store data in only two, and it can never serve a read or become leader because it does not have the rows.

Your instance configuration is really a choice of replica placement. A regional configuration puts read-write replicas in three zones of one region, so a zone can burn and writes continue on the remaining two voters. A multi-region configuration typically spans read-write replicas across two regions, a witness in a third, and read-only replicas wherever readers are; one region is the default leader region, and split leadership prefers to sit there. That placement is the largest lever on write latency you have -- every commit is a Paxos round trip from the leader to a quorum, so a leader region far from your writers taxes every write forever. See GCP multi-region architecture for the surrounding deployment concerns.

Paxos per split, two-phase commit across splits

Within a split, writes go through Paxos. One replica holds a time-bounded leader lease -- the paper describes leases on the order of ten seconds, renewed while the leader is healthy -- and the lease is what lets the leader answer reads from its own state without asking anyone, since nobody else can be leader for that interval. When a leader dies, writes to that split stall until the lease expires and a new one is elected, which is why a zone failure appears as seconds of write errors on some splits rather than a database outage. Stale reads keep working throughout.

A transaction touching two splits touches two Paxos groups, and Spanner layers classic two-phase commit on top. Each participating split's leader is a participant; one is coordinator. Prepare records and the commit decision are themselves written through Paxos in their own groups, which repairs 2PC's worst structural flaw: the usual objection is that a coordinator crash leaves participants blocked on locks indefinitely, but here the coordinator's state is replicated, so a crash elects a new leader and the protocol resumes.

What it does not repair is cost. A prepare round precedes the commit round, so latency is additive in the slowest participant rather than the average; locks are held across two rounds instead of one, widening the collision window and raising the abort rate superlinearly under contention; and the transaction now depends on every participant's leader staying available. This is why the schema advice below is not stylistic -- keeping a transaction single-split removes an entire protocol from the write path.

Read types: strong, bounded staleness, exact staleness

Spanner will not let you read without stating what you mean by "now". It is a correctness decision and also the cheapest latency lever in the system. All three read the same version history and take no locks; they differ only in which timestamp is chosen, and therefore in which replicas can answer.

Strong reads

Guaranteed to see everything committed before the read started. The default, and what you want for read-your-own-writes flows, for anything driving a subsequent write, and for anything a user is about to act on. The serving replica must establish that it is caught up past all prior commits, which in the general case means a round trip toward the leader -- in a multi-region configuration with a distant default leader, that is your read latency floor.

Bounded staleness

"Give me a snapshot no older than N seconds, and pick the timestamp yourself." Because Spanner chooses, it can choose one a nearby replica has already reached and answer locally with no leader round trip -- converting a cross-region read into a local one for dashboards, feeds and search results. The API constraint: bounded-staleness modes exist only for single-use read-only transactions, since the point is that Spanner picks per read.

Exact staleness

Pins the read to a specific timestamp, absolute or "now minus N". Deterministic and repeatable, so a multi-use read-only transaction can issue many queries against an identical snapshot -- what an export, reconciliation job or consistency check wants. The constraint is version garbage collection: old row versions survive only for the database's version retention period, and a read older than that fails outright rather than returning approximate data. Raising retention is deliberate and not free, since every superseded version stays live for the window.

The read-write transaction lifecycle, and why aborts are normal

Reads execute against the leader and take locks. Writes are not sent as issued -- the client library buffers mutations locally and ships them all at commit. So the leader sees a burst of reads, a pause of arbitrary length while your application thinks, then one commit carrying the whole write set, with the read locks held across that entire span.

Deadlock is avoided by wound-wait rather than detected. Each transaction carries its start timestamp as priority: if a younger transaction holds a lock an older one needs, the older wounds it and the younger aborts; the reverse case simply waits. Deadlock-free by construction, and a transaction that has already waited a long time becomes progressively harder to kill.

So ABORTED is not an error condition -- it is flow control, and it will happen under any real contention. The client libraries retry by re-executing the callback you gave them, which imposes the requirement most bugs here violate: the transaction body must be safe to run more than once.

# The callback is the retry unit. Everything inside it runs again on ABORTED.
def transfer(txn):
    row = txn.read_row("Accounts", ["a1"], ["Balance"])
    if row["Balance"] < 100:
        raise InsufficientFunds()          # fine: no external effect
    txn.update("Accounts", ["a1"], [row["Balance"] - 100])
    txn.update("Accounts", ["a2"], [...])

database.run_in_transaction(transfer)      # library retries with backoff
notify_user()                              # side effect goes OUTSIDE, after commit

The corollary of "locks held until commit": never hold a read-write transaction open across a user interaction, an external HTTP call, or a queue poll. Spanner eventually reaps idle transactions, but long before that it has been blocking every other writer touching those rows. When contention shows up in SPANNER_SYS.LOCK_STATS_TOP_MINUTE, the fix is a shorter transaction, not a bigger instance.

Interleaving: schema design as physical layout

Interleaving makes "stay inside one split" a schema-level guarantee. INTERLEAVE IN PARENT stores each child row physically adjacent to its parent, ordered by the parent's key prefix, and Spanner will not place a split boundary inside a parent's descendant hierarchy -- parent and children are one indivisible unit of placement.

CREATE TABLE Customers (
  CustomerId  STRING(36) NOT NULL,
  Name        STRING(MAX),
) PRIMARY KEY (CustomerId);

CREATE TABLE Orders (
  CustomerId  STRING(36) NOT NULL,   -- must lead with the parent key
  OrderId     STRING(36) NOT NULL,
  PlacedAt    TIMESTAMP,
) PRIMARY KEY (CustomerId, OrderId),
  INTERLEAVE IN PARENT Customers ON DELETE CASCADE;

You buy: a customer plus all their orders in one sequential read, a single-group commit with no 2PC when writing both, and a cascade delete that is a physical range operation. You pay: the child's key must lead with the parent's, foreclosing other orderings; and since the hierarchy cannot be split, one parent with unbounded children produces an oversized, unsplittable, un-rebalanceable split. Interleave where fan-out is bounded by something real -- orders per customer, line items per order. Do not interleave events under a tenant id that might generate billions.

Secondary indexes and what they cost transactionally

A Spanner secondary index is a real table with its own key ordering, its own splits and its own leaders -- which is the whole cost model. An insert into a base table with three indexes writes four independently placed key ranges, so four Paxos groups, so the write you thought was single-split is now a two-phase commit. Index count is write latency.

-- Covering index: satisfies the query without a back-join to the base table.
CREATE INDEX OrdersByPlacedAt ON Orders(PlacedAt) STORING (Status, Total);

-- Sparse index: skips rows where the key column is NULL.
CREATE NULL_FILTERED INDEX OrdersPendingReview ON Orders(ReviewRequestedAt);

-- Interleaved index: entries live in the parent's split, so writing the base
-- row and the index entry stays a single-group commit.
CREATE INDEX OrdersByPlacedAtPerCustomer ON Orders(CustomerId, PlacedAt),
  INTERLEAVE IN Customers;

STORING trades storage and write amplification for eliminating a back-join -- worth it on a hot query, wasteful applied indiscriminately. NULL_FILTERED is the right answer to "index the small set of rows in state X", where a dense index would be mostly empty entries. The interleaved index is the one people miss, and the only one of the three that reduces the number of Paxos groups a write touches rather than merely shrinking the index.

One trap deserves its own sentence: an index inherits none of your base table's key distribution. A UUID-keyed table spreads perfectly, and an index on CreatedAt over that same table concentrates every insert on one index split. The base table is fine and the database is still on fire.

Hotspots from monotonic keys

Because splits are contiguous key ranges, a monotonically increasing key sends every new row to the split holding the high end of the keyspace. Adding compute does not help -- the bottleneck is one leader -- and splitting does not either, because the new high split immediately becomes the next hot one. The general derivation for sorted, range-partitioned storage is in Cloud Bigtable: wide-column NoSQL at massive scale; these are the Spanner-specific mitigations.

A UUID primary key when nothing else is needed -- generated client-side, stored as STRING(36). Boring, and the default answer. Bit-reverse a sequential id when you must keep a dense integer key: reversing the bits scatters consecutive ids across the keyspace while staying lossless and reversible. Prefix with a hash-derived shard column when ordering within a group matters: a generated column such as MOD(FARM_FINGERPRINT(UserId), 64) as the leading key part spreads writes across sixty-four ranges while a single-user scan is still a bounded fan-out. Or just swap the key order -- (UserId, Timestamp) distributes where (Timestamp, UserId) concentrates.

Diagnosis is unusually good here: Key Visualizer renders key ranges against time as a heatmap, so a hotspot is a bright vertical band you read the offending key prefix straight off, not something you infer from aggregate CPU.

Schema changes run in the background

Spanner applies schema updates without downtime and without blocking reads or writes to the affected tables. The change is validated, assigned a timestamp, and applied as a versioned change: transactions before that timestamp see the old schema, transactions after see the new one. No maintenance window, no ALTER TABLE lock to plan around.

The nuance is that some changes carry a data obligation. Adding an index backfills every existing row; adding NOT NULL or narrowing a type validates every existing row. Those run as long-running operations that can take hours on a large table, consume real serving capacity while running, and fail cleanly -- a violating row rejects and rolls back the change rather than leaving it half applied. Two consequences: schema operations are serialized, so batch related DDL into one request instead of queueing statements behind each other; and treat a large index backfill as a scheduled capacity event, not instant metadata.

Sizing, autoscaling, and the operational surface

Capacity is provisioned as nodes, or in finer-grained processing units -- one node is one thousand processing units. That provisioned compute governs two ceilings at once: throughput, and how much data the instance may store. The storage ceiling is not soft; writes are rejected until you add compute, so storage growth rather than query load is what forces the scale-up on many quiet databases. Google publishes a high-priority CPU target that sits meaningfully below saturation, lower for multi-region than regional because a multi-region leader carries cross-region replication in the same budget -- look up the current figures rather than memorising them. The managed autoscaler moves capacity between a floor and ceiling you set; set the floor from your latency requirement, not your average load, because neither scale-up nor the split rebalancing that follows it is instantaneous.

The failure and latency characteristics operators actually meet, roughly by frequency: ABORTED under contention, normal and retried but a latency problem when transactions are long; p99 write latency dominated by cross-split 2PC, showing as a median-to-tail gap that extra compute never closes; a single hot split pinning one leader while instance CPU looks comfortable; FAILED_PRECONDITION on stale reads that fell outside version retention; and a latency shift after leadership changes while the new leader's cache is cold. All of these are diagnosed in SPANNER_SYS.QUERY_STATS_TOP_MINUTE, TXN_STATS_TOP_MINUTE and LOCK_STATS_TOP_MINUTE plus Key Visualizer -- instance-level CPU graphs mislead you on every one.

One decision that cannot be revisited: a database is created with either the GoogleSQL dialect or the PostgreSQL interface dialect, fixed for its lifetime. Choose against your portability requirements up front.

When Spanner is the right call

Spanner earns its cost when one of three things is true: you need more write throughput than one machine can give while keeping cross-row transactions and SQL; you need strong consistency across regions rather than a primary with async followers; or you need to survive zone and region loss without a failover a human runs and an RPO you explain afterwards.

If none hold, Cloud SQL is usually better and often by an order of magnitude in cost -- real Postgres or MySQL with its extension ecosystem, a shared regional disk that loses no committed write on zonal failover, and a far lower floor. You accept a single writer bounded by one machine and an RTO measured in the minute or two that engine start and crash recovery take. Spanner's floor is high enough that "we might need to scale later" is a weak reason to start here.

If you need neither cross-row transactions nor SQL, Bigtable gives better cost per operation at very high scale with the same sorted-keyspace model and no consensus on the write path. And Spanner is not only a product you rent: Google Cloud Storage's object metadata lives in a Spanner-backed store, which is where GCS's strong read-after-write consistency comes from. The machinery above is what makes a bucket listing correct.