Why architecture matters here
Repair mistakes are expensive. Running full repair everywhere every day saturates the network and hurts p99 latency. Never running it means tombstones stay past gc_grace and deleted data resurrects. Running with wrong concurrency causes cascading timeouts.
The architecture matters because the choice — full vs incremental, per-keyspace, per-subrange — is workload-dependent. Preview mode lets you measure without impact. Reaper automates scheduling.
Understanding the pieces means you set a policy that matches your data model and SLO.
The architecture: every piece explained
The top strip is the mechanism. Replica set is the nodes holding a given range. Merkle tree hashes the data per token range in a tree so replicas can compare hashes and locate diffs. Streaming session transfers the differing rows to reconcile. Repair scheduler — nodetool cron, or Reaper — coordinates when repairs run and where.
The middle row is variants. Full repair compares everything; expensive but authoritative. Incremental repair only compares data since last repair; much cheaper if maintained. Subrange repair narrows scope to specific token ranges — good for continuous rolling schedule. Preview mode computes the diff without streaming, useful for measuring drift. Anti-entropy service is the background actor that runs the sessions.
The lower rows are ops. Concurrent limits cap how many repairs run at once to avoid saturation. Observability tracks repair duration, data streamed, and errors. Ops layer schedules per keyspace based on data model, handles tombstone cleanup, and respects gc_grace_seconds.
End-to-end flow
End-to-end: Reaper runs incremental subrange repair across the cluster on a rolling schedule. It picks a keyspace, a small token range, and the responsible replicas. Each replica computes its Merkle tree for that range; hashes are compared; the diffs stream to reconcile. Repair completes in minutes for that range. Reaper moves to the next range. Full repair runs quarterly on smaller keyspaces. Preview mode is used to spot-check drift on the largest keyspace before scheduling full repair. Observability shows repair success rate and cluster health steady.