Cassandra Write and Read Path Architecture in Depth: coordinator, commit log, memtables, SSTables and reconciliation

The mental model: tokens, equal nodes and timestamps

Start with three facts. First, data is placed by token. The partitioner (Murmur3 by default) hashes the partition key to a 64-bit token, and the replication strategy walks the token ring to pick the replicas: with NetworkTopologyStrategy and {'dc1': 3, 'dc2': 3}, six nodes own every partition, three per data centre, spread across racks where possible.

Second, any node can coordinate any request. Third, every cell carries a write timestamp and conflicts are resolved by last write wins on it. That rule lets writes reach replicas in any order and still converge, and it is why clock skew can make a newer write lose.

Consistency is a per-request choice of how many replica responses to wait for. A read from R replicas overlaps a write acknowledged by W when R + W > RF. LOCAL_QUORUM for both, with RF 3, gives 2 + 2 > 3 inside one data centre, which is why it is the default choice for most applications.

The mental model: tokens, equal nodes and timestamps

Start with three facts. First, data is placed by token. The partitioner (Murmur3 by default) hashes the partition key to a 64-bit token, and the replication strategy walks the token ring to pick the replicas: with NetworkTopologyStrategy and {'dc1': 3, 'dc2': 3}, six nodes own every partition, three per data centre, spread across racks where possible.

Second, any node can coordinate any request. Third, every cell carries a write timestamp and conflicts are resolved by last write wins on it. That rule lets writes reach replicas in any order and still converge, and it is why clock skew can make a newer write lose.

Consistency is a per-request choice of how many replica responses to wait for. A read from R replicas overlaps a write acknowledged by W when R + W > RF. LOCAL_QUORUM for both, with RF 3, gives 2 + 2 > 3 inside one data centre, which is why it is the default choice for most applications.

The write path at the coordinator

A token-aware driver computes the token itself and sends the request straight to a replica, so the coordinator is usually also one of the owners and saves a network hop. The coordinator then does four things.

1. Resolve replicas. It computes the natural replicas for the token, plus any pending replicas if a node is joining or leaving, so the new owner does not miss writes during streaming.
2. Check liveness up front. Using the gossip failure detector, it counts live replicas. If fewer than the consistency level requires are up, it fails immediately with UnavailableException. Nothing was written.
3. Send the mutation to everyone. The write goes to all live replicas, not just enough to satisfy the consistency level. Local replicas get it directly. For each remote data centre the coordinator sends one copy to a single node there, which forwards it to the other replicas in its DC. This saves WAN bandwidth; the remote DC receives the write in the same fan-out, not through a separate asynchronous replication process.
4. Wait for the required acknowledgements. LOCAL_QUORUM waits for two of the three local replicas; remote acknowledgements still arrive but are not awaited. If the required acks do not arrive within write_request_timeout (2 seconds by default) the client gets WriteTimeoutException.

For each replica that was down when the write started, or that times out, the coordinator stores a hint in its own local hint files and replays it when gossip reports the replica back up. Hints are only kept for replicas that have been down less than max_hint_window (3 hours by default); after that the replica needs repair. A timeout is not a rollback: the write may already be on one replica, and it will spread from there through hints, read repair or repair. This is why idempotent writes and client-side timestamps matter for safe retries.

The write path on a replica

On each replica the mutation does two cheap things and returns.

1. Append to the commit log. The commit log is a write-ahead log shared by every table on the node, split into fixed-size segments. Its sync mode decides when the acknowledgement is safe. In the default periodic mode the replica acknowledges immediately and fsyncs every commitlog_sync_period (10 seconds by default), so a power loss on a node can lose up to that window of acknowledged writes on that node; the other replicas are the real durability. batch and group modes wait for fsync before acknowledging, trading latency for single-node durability.
2. Apply to the memtable. Each table has a memtable: a sorted in-memory structure keyed by partition and clustering key. Writes merge into it; nothing on disk is read or modified. That is why Cassandra writes are fast and roughly constant cost: no read-before-write, no in-place update.

When memtable memory crosses the cleanup threshold, the commit log reaches its total size limit, or an operator runs nodetool flush, a memtable is switched out and flushed to a new SSTable. An SSTable is a set of immutable files: the data file with compressed chunks, a partition index, a Bloom filter, a compression offset map, statistics (min and max timestamps and clustering bounds), and a table of contents. Once every memtable that had data in a commit log segment has flushed, that segment is recycled. On restart, the node replays only the segments that were not yet flushed.

Deletes and updates follow the same path. A delete writes a tombstone with a timestamp; an update writes new cells with a later timestamp. The old data stays on disk until compaction merges the SSTables and drops shadowed cells and tombstones older than gc_grace_seconds (10 days by default). Cassandra 5.0 adds trie-based memtables, the BTI SSTable format and the Unified Compaction Strategy, but the stock cassandra.yaml keeps the older defaults; the newer ones are enabled in the shipped cassandra_latest.yaml or by configuration.

The read path at the coordinator

The coordinator ranks the replicas with the dynamic snitch, which scores nodes by recent latency, and sends one full data request to the best replica and digest requests, which return only a hash of the result, to as many others as the consistency level needs. For LOCAL_QUORUM with RF 3, that is one data request and one digest request.

If the digests match, the coordinator returns the data. If they differ, it requests full data from the replicas it contacted, merges them cell by cell using write timestamps (tombstones included), and, because table option read_repair defaults to BLOCKING, writes the merged result back to the stale replicas before answering. Blocking read repair keeps monotonic quorum reads: once a quorum read has returned a value, a later quorum read cannot return an older one. The probabilistic read_repair_chance options were removed in 4.0.

speculative_retry (default 99p) sends an extra request to another replica when the first has not answered within the table's 99th percentile read latency. read_request_timeout (5 seconds by default) bounds the whole wait and produces ReadTimeoutException.

The read path on a replica

A partition can be spread over memtables and many SSTables, so a replica read is a merge. Unless the row cache (off by default) holds the partition, the replica reads the memtables, then filters SSTables cheapest check first:

1. Metadata. Statistics on min and max clustering keys and timestamps can skip SSTables that cannot contain the requested rows or cannot hold newer data than what is already found.
2. Bloom filter. An in-memory probabilistic set per SSTable answers "definitely not here" or "maybe". bloom_filter_fp_chance sets the false-positive rate (0.01 by default, 0.1 for LCS tables). A false positive costs an index lookup, never a wrong answer.
3. Partition index. In the classic BIG format, the key cache, an index summary and the partition index locate the partition's position in the data file. In the BTI format a trie index does the same job without the summary.
4. Data. The compression offset map translates the position to a compressed chunk (16 KiB by default), which is read and decompressed. Partitions with many rows also carry a row index so the replica can seek to the requested clustering range.

The replica then merges cells from all sources by timestamp and applies tombstones before replying. Every tombstone scanned costs memory and time; a query that crosses tombstone_warn_threshold (1,000) logs a warning, and one that crosses tombstone_failure_threshold (100,000) is aborted. The number of SSTables touched per read is the replica's real read amplification, and keeping it low is compaction's job. The deep read path article covers SSTable lookup internals in more detail.

Worked example: a sensor reading, written and read

Take a sensor table in two data centres, RF 3 in each, written and read at LOCAL_QUORUM.

CREATE KEYSPACE telemetry WITH replication =
  {'class': 'NetworkTopologyStrategy', 'dc1': 3, 'dc2': 3};

CREATE TABLE telemetry.readings (
  sensor_id  text,
  day        date,
  ts         timestamp,
  value      double,
  PRIMARY KEY ((sensor_id, day), ts)
) WITH CLUSTERING ORDER BY (ts DESC)
  AND compaction = {'class': 'TimeWindowCompactionStrategy',
                    'compaction_window_unit': 'DAYS', 'compaction_window_size': 1}
  AND default_time_to_live = 2592000;

// DataStax Java driver 4.x
PreparedStatement insert = session.prepare(
    "INSERT INTO telemetry.readings (sensor_id, day, ts, value) VALUES (?, ?, ?, ?)");

BoundStatement w = insert.bind("s-17", LocalDate.of(2026, 10, 5), Instant.now(), 21.4)
    .setConsistencyLevel(DefaultConsistencyLevel.LOCAL_QUORUM)
    .setIdempotent(true);          // safe for the driver to retry or speculate
session.execute(w);

SimpleStatement r = SimpleStatement.newInstance(
    "SELECT ts, value FROM telemetry.readings WHERE sensor_id = ? AND day = ? LIMIT 60",
    "s-17", LocalDate.of(2026, 10, 5))
  .setConsistencyLevel(DefaultConsistencyLevel.LOCAL_QUORUM);

The write: the driver hashes ('s-17', 2026-10-05) and sends it to replica A in dc1, which coordinates. A appends to its commit log and memtable, sends the mutation to B and C, and sends one copy to a node in dc2 that forwards it to its two peers. B acknowledges after about a millisecond; together with A's own write that is two of three, so the client gets success. C is in a long garbage collection pause and times out, so A writes a hint for C.

Thirty seconds later C is back and the hint is replayed. A read in between that reached B and C would see different digests, merge by timestamp, return B's reading and repair C first. Run TRACING ON in cqlsh to watch exactly these steps, with timings, for one query, and nodetool getendpoints telemetry readings <key> to see which nodes own a partition.

Failure modes

Operating the paths

Watch both paths separately. nodetool proxyhistograms shows coordinator latency, the view clients see; nodetool tablehistograms shows local latency, SSTables per read and partition size for one table. nodetool tpstats shows pending and blocked tasks per stage, and dropped messages, which mean replicas shed requests that had already timed out. nodetool compactionstats shows the compaction backlog.

• Run repair, preferably incremental or through a scheduler, more often than gc_grace_seconds; hints and read repair only fix what they happen to touch. See hinted handoff and read repair for each mechanism's limits.
• Choose the compaction strategy per table from its access pattern, using the compaction strategies guide.
• Alert on dropped messages, pending compactions, hints in flight and p99 coordinator latency, not just averages.

Trade-offs

The write path buys speed by deferring work: updates and deletes are cheap now and paid for later by merging reads and by compaction rewriting the same data. Lower consistency levels cut latency but give up read-your-writes. Last-write-wins makes conflicts cheap but has no notion of causality, so anything that needs compare-and-set must use lightweight transactions at SERIAL consistency and pay several extra round trips. The commit log article and the tombstones article go deeper on the two most common costs.

What to do next

1. Write down the RF and consistency level of every query path and confirm R + W > RF where you need read-your-writes.
2. Turn on TRACING ON for your hottest read and count the SSTables and tombstones it touches.
3. Check nodetool tablehistograms for SSTables per read above 4 to 5 at p99 and fix compaction first.
4. Confirm repair completes for every table within gc_grace_seconds.
5. Move the commit log to its own fast device and note the sync mode you rely on.
6. Mark idempotent statements in code and enable speculative execution only for them.
7. Bound every partition by design; find the largest with nodetool tablestats.