All 69 articles, sorted alphabetically
The CAP Theorem
The CAP Theorem explains why distributed systems cannot guarantee consistency, availability, and partition tolerance simultaneously. Understand the tr…
Read article →In Search of a Leader: Understanding Raft Consensus - AICassindra
In the world of distributed systems, getting a cluster of nodes to agree on something—like the order of log entries—is notoriously difficult.
Read article →API Gateway Patterns
Kong, Ambassador, AWS API Gateway. Auth (JWT validation), rate limits (per API key), routing (path-based, host-based, weighted), request/response tran…
Read article →Bulkhead Pattern
Isolate resources per feature. Prevent noisy neighbor.
Read article →Byzantine Fault Tolerance, in depth: the 3f+1 bound, PBFT's three phases, HotStuff's linear view change, and when BFT earns its cost
Byzantine fault tolerance from first principles: what the Byzantine fault model covers, why n must be at least 3f+1, partial synchrony and what it mea…
Read article →CAP Theorem + PACELC Extension, in depth: what the proof actually says, the latency cost of consistency when nothing is broken, and choosing per operation
CAP and its PACELC extension explained precisely: the Gilbert and Lynch definitions, three common misreadings, why replication forces a latency versus…
Read article →Chaos Engineering, in depth: injecting partitions, pauses, latency and clock faults into a distributed system and checking safety, not just uptime
Chaos engineering for distributed systems: a partial-fault model, tc netem, iptables, Toxiproxy and Chaos Mesh injection by layer, safety hypotheses c…
Read article →Circuit Breaker Pattern
Fail fast when dependency down. Prevent cascade failures.
Read article →Clocks in Distributed Systems, in depth: physical drift, wall versus monotonic time, timestamp ordering, Lamport clocks and choosing the right clock
A map of time in distributed systems: how hardware clocks drift and get synchronized, wall-clock versus monotonic clocks in code, why last-write-wins …
Read article →Consistent Hashing, in depth: Rings, Virtual Nodes and the Alternatives
Consistent hashing from first principles: ring lookup code, why one token per node is unbalanced and what virtual nodes cost, measured key movement, z…
Read article →CQRS, in depth
CQRS without event sourcing: the three levels of separation, command handlers with optimistic versions, why dual writes fail, outbox and CDC propagati…
Read article →CRDTs in Production, in depth: choosing data types, syncing with state vectors, storing and compacting history, evolving schemas and securing merges
What it takes to run conflict-free replicated data types in a real product: picking counter, register, set, map and sequence types by their surprising…
Read article →Anti-entropy -- healing divergence between replicas
Deep-dive on anti-entropy: the replica-divergence problem, background reconciliation, Merkle trees for efficient difference detection (O(1)/log(n)), r…
Read article →Bounded staleness architecture
Deep-dive on bounded-staleness consistency: version-based and time-based bounds, follower applied position and lag, read-path enforcement (serve, wait…
Read article →Causal consistency architecture
Deep-dive on causal consistency: the happened-before relation and session guarantees, dependency tracking with vector clocks, hold-until-ready applica…
Read article →Chain replication -- strong consistency with simple roles
Deep-dive on chain replication: the chain structure (head to tail), writes propagating down and committing at the tail, reads from the tail (committed…
Read article →Google Chubby, in depth: a coarse-grained lock service, its sessions, KeepAlives and cache invalidation, master failover, and the lessons that shaped ZooKeeper and etcd
How Google's Chubby lock service works, from the 2006 OSDI paper: why a lock service rather than a consensus library, the cell and master lease, …
Read article →Raft consensus architecture, in depth: deploying a Raft group, commit latency maths, placement, Multi-Raft and operations
Raft as a deployed component: where consensus belongs, the node pipeline from proposal to apply, commit latency from fsync and placement, sizing and m…
Read article →Raft Consensus, in depth: from the rules in the paper to a production node
Raft as you would implement and operate it: persistent state and fsync ordering, the vote and append handlers as code, why leaders only count replicas…
Read article →CRDT replication architecture
Deep-dive on CRDT replication: state-based vs op-based vs delta CRDTs, join-semilattice merges, OR-Sets and PN-counters, version vectors and dots, seq…
Read article →Distributed Hash Table Architecture in Depth
DHT architecture in depth: one id space for keys and nodes, virtual nodes, Chord fingers vs Kademlia k-buckets vs Pastry prefixes, join/leave converge…
Read article →etcd, in depth: the revisioned key space, Raft, transactions, watches and leases, and how to keep a cluster healthy
etcd from first principles: what it is for, how a member is built (gRPC, Raft, WAL, MVCC on bbolt), revisions and history, linearizable versus seriali…
Read article →Fencing tokens -- making distributed locks safe against pauses
Deep-dive on fencing tokens: the paused-lock-holder problem (GC/network/VM pauses past the lease), the monotonic fencing token issued per grant, resou…
Read article →Gossip-Based Membership, in depth: SWIM probing, suspicion and incarnations, piggybacked dissemination and Lifeguard
How clusters agree on who is alive without a coordinator: why all-to-all heartbeats do not scale, the SWIM probe cycle with indirect pings, suspicion …
Read article →Gray failure architecture
Deep-dive on gray failure: the degraded component whose shallow health check passes while real requests suffer, why differential observability is both…
Read article →Hinted handoff
Deep-dive on hinted handoff, the Dynamo-style mechanism that preserves write availability when a replica is temporarily down: a live custodian stores …
Read article →Hybrid Logical Clocks architecture
Deep-dive on Hybrid Logical Clocks: physical + logical combined, update rule, monotonicity, causality, bounded skew.
Read article →Distributed Leader Election Architecture in Depth
Leader election as a two-layer architecture: Raft elections inside etcd, PreVote and CheckQuorum, lease-based election for your own service, Kubernete…
Read article →Leases -- time-bounded exclusive rights
Deep-dive on distributed leases: the safe-exclusive-access need, the time-bounded grant, expiry (auto-release), renewal (by the live holder), contrast…
Read article →Merkle tree architecture
Deep-dive on Merkle trees for replica reconciliation: leaf hashes over key ranges, parent hashes up to a single root, root-then-descend comparison, dr…
Read article →Operational transformation architecture
Deep-dive on OT for collaborative editing: the operation model, the transformation function and TP1/TP2, the Jupiter central-server model, client pend…
Read article →Transactional outbox architecture
Deep-dive on the transactional outbox pattern: how writing the business change and the event to publish in one local database transaction eliminates t…
Read article →Paxos
How classical Paxos works: proposers, acceptors, learners, and the two-phase protocol that achieves consensus.
Read article →Phi-accrual failure detectors, in depth: per-peer state, Cassandra and Akka internals, pause guards and threshold calibration
How phi-accrual failure detectors work inside production systems: per-peer windows, the exponential model in Cassandra and the normal model in Akka, l…
Read article →Quorum systems
Deep-dive on quorum-based replication: the R+W>N overlap guarantee, tunable consistency levels, timestamp vs vector-clock versioning, read repair a…
Read article →Raft log replication
Deep-dive on Raft log replication: leader-based writes, log entries with term/index/command, AppendEntries replication, majority commit for fault tole…
Read article →Read repair and anti-entropy architecture
Deep-dive on the repair mechanisms behind leaderless eventual consistency: read repair on the coordinator's read path with the digest optimizatio…
Read article →Saga Pattern, in depth: compensations, the pivot step, a durable orchestrator and the isolation you give up
How the saga pattern keeps multi-service business transactions consistent without distributed locks: compensatable, pivot and retriable steps, orchest…
Read article →Split Brain, in depth: how two leaders happen, what they break, and how quorum, leases, fencing and STONITH stop them
Split brain explained from first principles: how network partitions, asymmetric links and process pauses create two active leaders, what dual writes d…
Read article →Total order broadcast architecture
Deep-dive on total order (atomic) broadcast: the agreement, total-order, validity, and integrity guarantees, equivalence to consensus, leader-sequence…
Read article →Vector clock architecture, in depth: causal delivery, consistent cuts and debugging distributed traces
Vector clocks as event-level infrastructure: a clock layer between application and transport, causal broadcast with a hold-back queue, recovering cons…
Read article →Vector Clocks in Depth: Tracking Causality, Detecting Concurrent Writes and Keeping the Metadata Bounded
How vector clocks capture happens-before exactly: first principles, the update and comparison rules with Python code, a worked three-process example, …
Read article →Replication watermarks architecture
Deep-dive on watermarks in a replicated log: the high-water mark as the committed, safe-to-read boundary computed from the minimum offset replicated t…
Read article →ZooKeeper architecture, in depth: znodes, the ensemble, Zab, sessions, watches, consistency guarantees and how to run it
How Apache ZooKeeper works from the inside: the znode data model and node types, leaders, followers and observers, the Zab atomic broadcast and zxids,…
Read article →Distributed Locking Patterns, in depth: database locks, Redis, Redlock, ZooKeeper and etcd, and fencing
A catalogue of distributed locking patterns and how to choose: efficiency versus correctness locks, why leases need fencing tokens, database row and a…
Read article →Distributed Tracing with OpenTelemetry, in depth: causality, the propagation contract, trust boundaries, clock skew and trace completeness
Distributed tracing as a distributed-systems problem: the causal span model, W3C traceparent, tracestate and baggage, where propagation breaks, durabl…
Read article →Distributed Transactions, in depth: 2PC and 3PC as state machines, termination rules and a crash-point model check
Atomic commitment from first principles: 2PC and 3PC as participant state machines, why the prepared state blocks, how the 3PC pre-commit round and te…
Read article →Calvin, in depth: deterministic distributed transactions without two-phase commit
How Calvin agrees on transaction inputs before execution: 10 ms sequencer epochs, Paxos or async input replication, deterministic locking, active and …
Read article →Distributed Transactions in Practice, in depth: choosing between co-location, XA, distributed SQL, sagas and idempotent retries
A practical guide to making changes atomic across machines and services: the decision ladder, how two-phase commit and XA fail in production, how Perc…
Read article →The Dynamo Paper, in depth: how Amazon's always-writable key-value store composed six techniques, and what its descendants kept
A first-principles reading of Amazon's 2007 Dynamo paper as a whole system: the requirements that forced each design choice, how consistent hashi…
Read article →Event Streaming Patterns, in depth: notification vs state transfer, fan-out, outbox, compacted tables, idempotent projections and dead letters
A practical catalogue of event streaming patterns on a partitioned log: event notification versus state transfer, fan-out with consumer groups, outbox…
Read article →Failure Detectors and Phi Accrual, in depth: completeness, accuracy, the detector classes, and building timeout, phi and SWIM detectors that behave in production
How distributed failure detectors really work: why they can only suspect, completeness and accuracy, the Chandra-Toueg classes, detection time and mis…
Read article →Google Spanner and TrueTime, in depth: commit wait and its proof, timestamp rules, safe time for replica reads and what to copy without atomic clocks
How Spanner uses TrueTime for external consistency: the interval API and the 2012 uncertainty figures, the Start and Commit Wait rules with the four-s…
Read article →Gossip Protocol for Cluster Membership, in depth: epidemic dissemination, versioned state and digest reconciliation
How gossip spreads cluster membership state: push, pull and push-pull epidemics with a simulation and measured round counts, versioned endpoint state …
Read article →What Jepsen Taught Us, in depth: histories, nemeses, checkers and the failure patterns that keep coming back
What a decade of Jepsen analyses teaches about distributed systems: how a Jepsen test is built, why indeterminate operations matter, how Knossos and E…
Read article →Kubernetes Operator Pattern, in depth: level-triggered reconcile loops, CRD design, finalizers and running operators safely
How Kubernetes operators work from first principles: informers, deduplicating workqueues and level-triggered reconciliation, spec and status design wi…
Read article →Leader election algorithms, in depth: bully, ring, quorum-and-term and lease-based election compared, with code and fencing
A comparative guide to leader election algorithms: the safety and liveness problem, the bully algorithm, ring algorithms (LCR, Chang-Roberts, Hirschbe…
Read article →Linearizability vs Sequential Consistency, in depth: histories, real-time order, a working checker, composability, cost, and where real systems land
A first-principles comparison of linearizability and sequential consistency: operation histories, the one difference (real-time order), worked histori…
Read article →Quorum Systems Explained, in depth: the intersection property, majorities, weighted votes, grids, Flexible Paxos and Byzantine quorums
Quorum systems from first principles: why intersection is the only property that matters, majority and weighted voting, grid quorums and the load vers…
Read article →Quorum Writes and Reads, in depth: the coordinator request path, what a timeout really means, and why R plus W over N is not linearizable
What happens on a quorum write and read in a Dynamo-style store: the coordinator, digest reads, blocking read repair, hinted handoff and sloppy quorum…
Read article →Raft Consensus Algorithm, in depth: the five safety properties as code, deterministic simulation, and the bugs real implementations shipped
How to know a Raft implementation is correct: the algorithm in brief, the paper's five safety properties written as an executable checker, a dete…
Read article →Raft Consensus Intuition, in depth: deriving every rule by breaking a simpler design
Build Raft from first principles: start with a primary and a backup, break it with partitions and crashes, and add one rule per failure until you reac…
Read article →Retry, Exponential Backoff and Jitter, in depth: what to retry, jitter variants, retry budgets, deadlines and retry amplification
How to retry safely in distributed systems: classifying retryable errors, idempotency keys, exponential backoff with full, equal and decorrelated jitt…
Read article →Saga: Choreography vs Orchestration, in depth
Saga choreography versus orchestration in depth: one checkout saga built both ways in code, the cost of adding a step, how each style detects a stuck …
Read article →Istio vs Linkerd, in depth: sidecar, ambient and micro-proxy data planes, identity, policy, upgrades and choosing a mesh
Choosing and operating a service mesh: Istio sidecar and ambient mode (ztunnel, HBONE, waypoints) versus Linkerd's Rust micro-proxy, control plan…
Read article →Sharding Strategies, in depth: choosing a shard key from your workload
How to choose a sharding scheme from evidence: partition keys versus partition functions, the balls-into-bins imbalance bound and heavy-tailed keys, a…
Read article →Two-Phase Commit, in depth: building a coordinator, recovering in-doubt branches and testing every crash point
A hands-on guide to two-phase commit: a runnable coordinator over PostgreSQL prepared transactions, a fenced recovery loop, a crash-point test matrix,…
Read article →Two-Phase Commit Revisited, in depth
Two-phase commit at the level of log records: the basic protocol, presumed abort and presumed commit, read-only and one-phase optimisations, the block…
Read article →YugabyteDB, in depth: DocDB tablets, Raft per shard, hybrid time transactions and multi-region design
How YugabyteDB works: the YSQL query layer over DocDB, hash and range sharding into tablets, Raft replication with leader leases, hybrid time and dist…
Read article →