Why architecture matters here
Cassandra architecture matters because every operational decision — replication factor, consistency level, compaction strategy, repair cadence — is a trade-off between latency, availability, and consistency. Get one wrong and the cluster behaves poorly in ways that feel like bugs but are actually configuration mismatches.
Consider consistency level. LOCAL_QUORUM for reads and writes gives you strong-ish consistency within a DC at low latency. Change one to ONE and you get faster reads but stale answers under partition. Change one to ALL and you can't survive a node failure. Every application team makes this decision, often without understanding the underlying math. The architecture is what tells you why each setting behaves the way it does.
Compaction is another lever. Size-tiered (STCS) is good for write-heavy workloads. Leveled (LCS) is good for read-heavy workloads with predictable partition sizes. Time-window (TWCS) is best for time-series data. The wrong choice means either massive read amplification or a compaction backlog that never catches up. Getting compaction right is one of the most impactful operational choices.
The architecture: every piece explained
Read the diagram from top to bottom.
Client Driver. Modern drivers (Java, Python, Go) are token-aware — they compute the partition token client-side and send the request to a replica for that token. This eliminates one hop compared to naive drivers. DC-aware routing prefers replicas in the client's local DC. Both of these are essential for good tail latency.
Coordinator Node. Any node in the cluster can act as a coordinator. It receives the request and orchestrates the write or read across the replicas responsible for the partition. Coordination is stateless; failure of the coordinator during a request means the client retries against another node.
Consistency Level. Per-query knob that says how many replicas must acknowledge before the operation returns. ONE = fastest, weakest. QUORUM = majority = strong. LOCAL_QUORUM = majority within the local DC = good default in multi-DC setups. ALL = every replica = strongest but brittle.
Write Path. On each replica, the write appends to the commit log (for durability) and inserts into the memtable (for reads). Both operations are fast; a well-tuned commit log absorbs 50,000+ writes per second per node. Writes never modify existing data on disk; they only append.
Read Path. Reads consult the row cache first (if configured), then the memtable, then any SSTables that might contain the partition. Bloom filters let the read skip SSTables that definitely do not contain the partition; without them, read amplification would be crushing.
Memtable. An in-memory, sorted map of writes per column family. When the memtable exceeds a threshold, it flushes to disk as an SSTable. The commit log allows recovery of unflushed writes if the node crashes.
SSTables. Immutable, sorted, indexed files on disk. Each partition can be spread across many SSTables. Reads may have to consult several to reconstruct the full state; compaction is what keeps that count manageable.
Bloom Filter. A probabilistic set membership check per SSTable. Given a partition key, the bloom filter says "definitely not here" or "possibly here." False positives cost a disk seek; false negatives are impossible.
Compaction. Background process that merges SSTables together, dropping tombstones and overlapping data. STCS merges similar-sized SSTables; LCS keeps SSTables in size-bounded levels; TWCS keeps SSTables partitioned by time window. Compaction is where the "log-structured" name earns its keep.
Anti-entropy Repair. Periodic process that compares data across replicas using Merkle trees and streams differences. Necessary because eventual consistency accumulates small deltas over time (hinted handoff timeouts, missed writes). Repair converges replicas.
Multi-DC Replication. Each DC has its own set of replicas. Writes propagate asynchronously between DCs. LOCAL_QUORUM at each DC gives strong consistency locally while accepting eventual consistency between DCs.
End-to-end write and read flow
Trace a write. The client wants to insert user_id=42, name='Alice' with LOCAL_QUORUM in a 3-replica-per-DC configuration.
The driver computes the token for user_id=42 and picks the closest replica for that token as the coordinator. The coordinator sends the write to all three replicas in the local DC in parallel. Each replica appends to its commit log, inserts into its memtable, and acknowledges. When two of the three acknowledge (quorum), the coordinator returns success to the client. The third replica's write eventually lands; if that replica was down, a hinted handoff stores the write on another node until the original comes back.
Meanwhile the write propagates asynchronously to the remote DC's replicas. Reads from the remote DC after the propagation delay will see Alice; reads before will not.
Trace a read. Client reads user_id=42 with LOCAL_QUORUM. The coordinator picks two of the three local replicas and sends read requests to both. Each consults its memtable (fast), then the row cache (if enabled), then bloom-filters the SSTables to find candidates, then reads and merges the candidates. The coordinator compares the two responses; if they differ (implying stale data on one), it triggers a read repair to bring the stale replica up to date. It returns the freshest value.
Compaction is running in the background all the time. When memtables flush, new SSTables land; compaction merges them into larger ones over time, dropping tombstones and overlapping columns. A repair also runs periodically, comparing Merkle trees between replicas and streaming any differences.