Cassandra Multi-DC Architecture in Depth

What multiple datacenters buy, and what they do not

Multi-DC buys three things. Latency locality: a user in Frankfurt reads and writes against Frankfurt replicas instead of paying a transatlantic round trip, typically 70 to 100 ms, on every request. Survival of a regional failure: with a full copy elsewhere, a lost region becomes a traffic shift rather than a restore from backup. Workload isolation: a DC that receives replicated data but no application traffic can run analytics scans or Spark jobs without hurting the latency of the serving DCs.

It does not buy a backup. A DELETE, a TRUNCATE or a bad batch job replicates to every DC within milliseconds, so snapshots and point-in-time backups are still required. It also does not buy synchronous cross-region consistency unless you ask for it with a consistency level that waits on the WAN, and pay for it in latency on every request.

What multiple datacenters buy, and what they do not

Multi-DC buys three things. Latency locality: a user in Frankfurt reads and writes against Frankfurt replicas instead of paying a transatlantic round trip, typically 70 to 100 ms, on every request. Survival of a regional failure: with a full copy elsewhere, a lost region becomes a traffic shift rather than a restore from backup. Workload isolation: a DC that receives replicated data but no application traffic can run analytics scans or Spark jobs without hurting the latency of the serving DCs.

It does not buy a backup. A DELETE, a TRUNCATE or a bad batch job replicates to every DC within milliseconds, so snapshots and point-in-time backups are still required. It also does not buy synchronous cross-region consistency unless you ask for it with a consistency level that waits on the WAN, and pay for it in latency on every request.

Deployment patterns

Most deployments follow one of three patterns, and the right one depends on where writes originate.

Geo-homed active-active is the most common choice for user-facing systems. Most rows have a natural owner, such as the user's region, so routing writes for that row to one DC removes nearly all conflicts while every region still serves fast local reads. The keyspace definition is the same for every pattern; only the application's routing differs.

CREATE KEYSPACE payments WITH replication = {
  'class': 'NetworkTopologyStrategy',
  'us-east': 3,
  'eu-west': 3,
  'analytics': 2      -- isolation DC: replicas, no application clients
};

The write path across datacenters

Follow a write from an application in us-east at LOCAL_QUORUM. The driver is configured with us-east as its local DC and is token-aware, so it sends the request to a local node that owns the partition, which acts as coordinator. The coordinator sends the mutation to all three local replicas directly. For each remote DC it sends one copy of the mutation to a single replica there, which relays it to the other replicas in its own DC. This forwarding means a write crosses the WAN once per remote DC rather than once per remote replica, which is why cross-DC bandwidth scales with the number of DCs rather than with the total RF. Internode compression between DCs is on by default, which helps further.

The consistency level controls only how many acknowledgements the coordinator waits for, not how many replicas receive the write. Every replica in every DC is sent the mutation regardless of level. At LOCAL_QUORUM with RF 3 the coordinator returns success once two us-east replicas have written it to their commit log and memtable. The eu-west copies are usually applied tens of milliseconds later. If a replica is down or times out, the coordinator stores a hint for it and replays the hint when the node returns, as long as the outage is shorter than the hint window, which is three hours by default. Longer gaps, and anything hints miss, are closed by anti-entropy repair.

Consistency arithmetic across datacenters

Quorum arithmetic is per DC for the LOCAL_ levels and cluster-wide otherwise. For a keyspace with RF 3 in each of two DCs (six replicas in total):

Two consequences catch teams out. First, QUORUM across two equal DCs is worse than it sounds: it always needs at least one remote ack, so it pays WAN latency, and it fails outright when either DC is lost. With two DCs, the usable choices are LOCAL_QUORUM for availability or EACH_QUORUM for writes that must be durable in both regions before the client is told. Second, read-your-writes holds only inside a DC. A LOCAL_QUORUM write in us-east followed by a LOCAL_QUORUM read in us-east always overlaps, because two of three plus two of three must share a replica. The same read in eu-west can return the old value until replication catches up. If a user's requests can land in either region, pin each session to one DC. Cassandra consistency levels covers single-DC behaviour in depth.

Lightweight transactions: SERIAL vs LOCAL_SERIAL

Lightweight transactions (INSERT ... IF NOT EXISTS, UPDATE ... IF balance = 10) run a Paxos round before the write. Their serial consistency level decides who takes part. With SERIAL, the default, a quorum of replicas across all DCs participates, so every LWT pays several WAN round trips. Cassandra 4.1 introduced an improved Paxos implementation that cuts the number of round trips, but they still cross the WAN. With LOCAL_SERIAL, only the local DC's replicas take part, so the LWT is fast but linearizable only among clients in that DC.

That last point is subtle. If an application in us-east and one in eu-west both run INSERT INTO usernames ... IF NOT EXISTS at LOCAL_SERIAL for the same name at the same moment, both can succeed, and replication later resolves the two inserts by timestamp. Use SERIAL for global uniqueness, or make the geo-homed pattern do the work: route all LWTs for a partition to that partition's home DC and use LOCAL_SERIAL there.

Active-active writes and last-write-wins

Without LWTs, Cassandra resolves concurrent writes cell by cell using write timestamps. Each column value carries a microsecond timestamp, set by the client driver (the default in modern drivers) or by the coordinator. During reads, compaction and repair, the value with the highest timestamp wins. If us-east updates email and eu-west updates phone on the same row, both changes survive because they touch different cells. If both update email, the later timestamp wins everywhere and the other write disappears without an error.

Clock skew turns this from a theoretical concern into real data loss. A client whose clock runs two seconds fast writes values that beat any honest write for the next two seconds, and a delete issued from a slow clock can fail to remove data that appears newer. Keep NTP or a cloud time service tightly bounded on every client and node, and alert on offset. Then design around last-write-wins: model changes as appended rows, such as one row per ledger entry instead of one mutable balance column, so concurrent writers add rows instead of overwriting each other. Counters are not idempotent under retries, which makes them a poor fit for anything that must reconcile.

Worked example: losing a datacenter

A payments platform runs geo-homed active-active on us-east and eu-west, RF 3 each, at LOCAL_QUORUM. Normal replication lag is around 80 ms. At 14:02 the us-east region loses network connectivity entirely.

1. Detect, then decide. Health checks fail at 14:02:30. The runbook requires two independent signals and a human decision before a full region failover, because a 40-second network blip followed by an automated failover costs more than the blip did.
2. Shift traffic. At 14:06 the global load balancer sends US users to the eu-west application tier, whose drivers already use eu-west as their local DC. US users now pay about 80 ms extra per request but stay up. RTO in practice is detection plus decision plus DNS or load-balancer propagation, here about four minutes.
3. Account for RPO. Writes that us-east acknowledged in roughly the last 80 ms, plus any backlog of hints that were queued because the WAN was already degraded, never reached eu-west. They are not lost while the us-east disks survive, but they are invisible until the region returns. If that window is unacceptable for a table, write it at EACH_QUORUM and accept the latency.
4. Watch the hint window. eu-west coordinators now store hints for every us-east replica. After three hours they stop, and from then on only repair can reconcile the regions.
5. Return carefully. us-east comes back at 19:30, after the hint window. Before routing any US traffic home, run a full repair of the affected keyspaces across both DCs so the returning replicas learn about writes and deletes made in eu-west. If a region were down for longer than gc_grace_seconds (ten days by default), its nodes must not simply rejoin: tombstones elsewhere may have been purged, and the old replicas would bring deleted data back to life. Replace or rebuild those nodes instead.

The driver configuration below keeps each application pinned to its own DC. The Python driver's DCAwareRoundRobinPolicy uses no remote hosts by default, and the Java driver 4.x does not fail over across DCs by default either. Both choices are deliberate: failover is a traffic decision made above the driver, and silently sending LOCAL_QUORUM to a remote coordinator would change both latency and the meaning of "local".

import os
from cassandra import ConsistencyLevel
from cassandra.cluster import Cluster, ExecutionProfile, EXEC_PROFILE_DEFAULT
from cassandra.policies import DCAwareRoundRobinPolicy, TokenAwarePolicy

LOCAL_DC = os.environ["CASSANDRA_LOCAL_DC"]          # "us-east" or "eu-west"

profile = ExecutionProfile(
    load_balancing_policy=TokenAwarePolicy(DCAwareRoundRobinPolicy(local_dc=LOCAL_DC)),
    consistency_level=ConsistencyLevel.LOCAL_QUORUM,
    serial_consistency_level=ConsistencyLevel.LOCAL_SERIAL,  # home-DC LWTs only
    request_timeout=2.0,
)
cluster = Cluster(contact_points=LOCAL_SEEDS,
                  execution_profiles={EXEC_PROFILE_DEFAULT: profile})
session = cluster.connect("payments")

Operating a multi-DC cluster

• Repair on a schedule. Every token range must be repaired more often than gc_grace_seconds. Use a scheduler such as Cassandra Reaper, or the incremental repair that became reliable in 4.0, and alert when a range has not been repaired in time.
• Watch hints and dropped mutations. Rising hint counts mean a replica or WAN link is unhealthy; dropped mutations mean writes are timing out on replicas. Both should page someone before the hint window expires. Hinted handoff explains the mechanics and limits.
• Budget the WAN. Steady-state cross-DC traffic is roughly write bytes times the number of remote DCs, before compression. Streaming for repair and rebuild comes on top, and Cassandra can throttle inter-DC streaming so it does not starve replication.
• Keep topology honest. The snitch must report the right DC and rack for every node; a node that joins with the wrong DC name silently changes replica placement.
• Rehearse failover. Run a game day each quarter that moves real traffic to the other region and back, including the repair step. A failover that has never been tested is a hope rather than a plan.

What to do next

1. Write down your pattern (active-passive, geo-homed or shared rows) and which tables can tolerate last-write-wins.
2. Grep the code for QUORUM and SERIAL without the LOCAL_ prefix, and justify each use or change it.
3. Pin each application instance to a local DC in the driver, and keep cross-DC failover in the traffic layer.
4. Measure replication lag and hint backlog per DC, and state your RPO as a number.
5. Confirm every range is repaired within gc_grace_seconds, and write the "region back after the hint window" repair step into the runbook.
6. Schedule a failover game day and time each step against your RTO target.