Distributed Systems

Understanding the CAP Theorem

Gossip Protocol - AiCassindra

Gossip Protocol - AiCassindra

Understanding LSM Trees: The Write-Optimized Storage Engine

Understanding LSM Trees: The Write-Optimized Storage Engine

In Search of a Leader: Understanding Raft Consensus - AICassindra

API Gateway Patterns

Kong + Ambassador + AWS API Gateway. Auth + routing + rate limits.

Bulkhead Pattern

Isolate resources per feature. Prevent noisy neighbor.

Byzantine Fault Tolerance

BFT in blockchains, critical control systems, and where it isn't worth the overhead.

CAP Theorem + PACELC Extension

Consistency-Availability-Partition trade-off. Modern PACELC.

Chaos Engineering

Netflix Chaos Monkey. Test resilience by breaking things.

Circuit Breaker Pattern

Fail fast when dependency down. Prevent cascade failures.

Clocks in Distributed Systems

Wall clocks lie. Logical clocks and what they buy you.

CockroachDB

Range-partitioned + Raft + HLC. PostgreSQL-compatible.

Consensus Protocols Compared

Why we have multiple consensus protocols and how to pick one.

Consistent Hashing

Add/remove nodes with minimal re-shuffling. Foundation of NoSQL.

CQRS

Separate write + read models. Different databases + optimization.

CRDTs

Eventually consistent without coordination. Counters, sets, maps.

CRDTs

Data structures that merge concurrent updates deterministically without coordination.

CRDTs in Production

When commutative types beat consensus.

Anti-entropy -- healing divergence between replicas

Deep-dive on anti-entropy: the replica-divergence problem, background reconciliation, Merkle trees for efficient difference detection (O(1)/log(n)), r…

Bounded staleness architecture

Deep-dive on bounded-staleness consistency: version-based and time-based bounds, follower applied position and lag, read-path enforcement (serve, wait…

Causal consistency architecture

Deep-dive on causal consistency: the happened-before relation and session guarantees, dependency tracking with vector clocks, hold-until-ready applica…

Chain replication -- strong consistency with simple roles

Deep-dive on chain replication: the chain structure (head to tail), writes propagating down and committing at the tail, reads from the tail (committed…

Google Chubby

What Chubby is, how it uses Paxos for consensus, and how it inspired ZooKeeper.

Raft consensus architecture

Deep-dive on Raft consensus: roles, log replication, commit index, snapshots, joint-consensus membership, and the operational surface.

Raft Consensus

How Raft achieves distributed consensus through leader election, log replication, and safety guarantees.

Consistent Hashing

How consistent hashing distributes keys across nodes such that adding or removing a node moves only 1/N of keys.

CRDT replication architecture

Deep-dive on CRDT replication: state-based vs op-based vs delta CRDTs, join-semilattice merges, OR-Sets and PN-counters, version vectors and dots, seq…

CRDTs

How CRDTs enable strong eventual consistency by having replicas that always merge cleanly.

Distributed Hash Table Architecture in Depth

A 2500-word walkthrough of DHT architecture: key + node id space, Chord finger tables, Kademlia k-buckets, lookup, replication, churn, stabilization, …

etcd

What etcd is, how it uses Raft consensus, and its role in Kubernetes.

Fencing tokens -- making distributed locks safe against pauses

Deep-dive on fencing tokens: the paused-lock-holder problem (GC/network/VM pauses past the lease), the monotonic fencing token issued per grant, resou…

Gossip Protocols

How gossip protocols spread information across cluster members through random peer communication, and how they enable membership and failure detection…

Gossip protocol architecture

Deep-dive on gossip protocols: SWIM, peer selection, rumor spreading, suspicion, indirect ping, anti-entropy, convergence.

Gossip-Based Membership

How gossip protocols track cluster membership without central coordination.

Gray failure architecture

Deep-dive on gray failure: the degraded component whose shallow health check passes while real requests suffer, why differential observability is both…

Hinted handoff

Deep-dive on hinted handoff, the Dynamo-style mechanism that preserves write availability when a replica is temporarily down: a live custodian stores …

Hybrid Logical Clocks (HLC)

How HLC combines physical wall clock with logical Lamport component for better ordering.

Hybrid Logical Clocks architecture

Deep-dive on Hybrid Logical Clocks: physical + logical combined, update rule, monotonicity, causality, bounded skew.

Leader Election

The leader election problem, the algorithms (bully, ring, Paxos/Raft-based), and how systems like Kubernetes and ZooKeeper implement it.

Distributed Leader Election Architecture in Depth

A 2500-word walkthrough of leader election in modern distributed systems: heartbeat, election timeout, PreVote, RequestVote, majority, split vote, and…

Leases -- time-bounded exclusive rights

Deep-dive on distributed leases: the safe-exclusive-access need, the time-bounded grant, expiry (auto-release), renewal (by the live holder), contrast…

Merkle tree architecture

Deep-dive on Merkle trees for replica reconciliation: leaf hashes over key ranges, parent hashes up to a single root, root-then-descend comparison, dr…

Operational Transform

How Operational Transform enabled Google Docs-style collaborative editing before CRDTs.

Operational transformation architecture

Deep-dive on OT for collaborative editing: the operation model, the transformation function and TP1/TP2, the Jupiter central-server model, client pend…

Transactional outbox architecture

Deep-dive on the transactional outbox pattern: how writing the business change and the event to publish in one local database transaction eliminates t…

Paxos

How classical Paxos works: proposers, acceptors, learners, and the two-phase protocol that achieves consensus.

Phi-accrual failure detectors

Deep-dive on phi-accrual failure detection: why fixed timeouts fail, the sampling window and gap-distribution estimator, how phi maps silence to log-s…

Quorum

How quorum-based systems (N, R, W) balance consistency, availability, and performance.

Quorum systems

Deep-dive on quorum-based replication: the R+W>N overlap guarantee, tunable consistency levels, timestamp vs vector-clock versi…

Raft log replication

Deep-dive on Raft log replication: leader-based writes, log entries with term/index/command, AppendEntries replication, majority commit for fault tole…

Read repair and anti-entropy architecture

Deep-dive on the repair mechanisms behind leaderless eventual consistency: read repair on the coordinator's read path with th…

Saga pattern architecture

Deep-dive on saga pattern: choreography vs orchestration, local transactions, compensations, timeouts, idempotency, semantic isolation.

Saga Pattern

How the saga pattern implements long-running transactions via a sequence of local transactions with compensating actions on failure.

Split Brain

How split brain occurs, its consequences (dual writes, data corruption), and prevention (quorum, fencing).

Total order broadcast architecture

Deep-dive on total order (atomic) broadcast: the agreement, total-order, validity, and integrity guarantees, equivalence to consensus, leader-sequence…

Two-Phase Commit

How two-phase commit (2PC) achieves atomic distributed transactions, its failure modes, and why it's rarely the right choice at s…

Two-phase commit architecture

Deep-dive on 2PC: coordinator + participants, prepare + commit phases, blocking failure mode, alternatives (sagas, LWT).

Vector clock architecture

Deep-dive on vector clocks: node counters, increment/merge, version vectors, conflict detection, resolution, GC, alternatives.

Vector Clocks

How vector clocks record happened-before relationships in distributed systems, and how they enable causal consistency.

Replication watermarks architecture

Deep-dive on watermarks in a replicated log: the high-water mark as the committed, safe-to-read boundary computed from the minimum offset replicated t…

Apache ZooKeeper

How ZooKeeper provides distributed coordination via a hierarchical namespace and ZAB consensus.

ZooKeeper architecture

Deep-dive on Apache ZooKeeper: ensemble, ZAB protocol, znodes, ephemeral/sequential flags, watches, sessions, use cases.

Distributed Locking Patterns

How to safely coordinate exclusive access across nodes with Redis, ZK, or lease-based systems.

Distributed Tracing with OpenTelemetry

Trace requests across services. Standard instrumentation.

Distributed Transactions

Two-phase and three-phase commit. Blocking + failure modes.

Calvin

Yale 2012. Deterministic ordering enables cheap distributed transactions.

Distributed Transactions in Practice

2PC, Saga, TCC — real-world patterns and trade-offs.

The Dynamo Paper

2007 Amazon paper. Shaped modern NoSQL: Cassandra, Riak, DynamoDB.

Event Sourcing

State derived from event stream. Time travel + audit + CQRS.

Event Streaming Patterns

Common Kafka + Kinesis usage patterns.

Failure Detectors and Phi Accrual

Why fixed timeouts are wrong and what Cassandra Akka use instead.

Google Spanner + TrueTime

Globally-distributed SQL with external consistency via GPS + atomic clocks.

Gossip Protocol for Cluster Membership

How rumors spread through a cluster to keep membership and state eventually consistent.

Gossip Protocols

Anti-entropy, push/pull, rumor mongering. Cassandra + Consul + others.

What Jepsen Taught Us

A decade of finding consistency bugs in production databases.

Kubernetes Operator Pattern

Custom controllers for domain-specific resources. Automate ops.

Leader Election Algorithms

Three algorithms for picking a leader in a distributed cluster, with tradeoffs.

Linearizability vs Sequential Consistency

The two most important consistency models, side by side.

Transactional Outbox Pattern

Ensure DB write + event publish are atomic. Foundation of event-driven microservices.

Paxos

Lamport 1998. Basic Paxos + Multi-Paxos + variants.

Paxos Algorithm

How Paxos achieves agreement across N nodes even when some fail, phase by phase.

Quorum Systems Explained

Sloppy quorum NWR and what R+W>N actually means.

Quorum Writes and Reads

How to configure Dynamo-style storage for consistency without linearizability.

Raft Consensus Algorithm

Raft's leader-based design that makes distributed agreement approachable.

Raft Consensus

Leader election + log replication + safety. Etcd/CockroachDB/TiDB use it.

Raft Consensus Intuition

Leader election, log replication, and why it's easier than Paxos.

Retry + Exponential Backoff + Jitter

Retry transient failures. Backoff prevents thundering herd. Jitter randomizes.

Saga: Choreography vs Orchestration

Deep dive into the two saga variants. When to use which.

Saga Pattern

Sequence of local transactions + compensating actions.

Service Discovery

Find services dynamically. Registration + lookup.

Service Mesh

Sidecar proxies. mTLS + retries + telemetry. Trade-offs.

Sharding Strategies

Split data across nodes. Trade-offs per strategy.

Two-Phase Commit

The classic 2PC protocol, its guarantees, and why it's rarely used in modern systems.

Two-Phase Commit Revisited

Where 2PC fits and where it doesn't in 2026.

Vector Clocks

Determine happens-before. Alternative to physical timestamps.

Vector Clocks

How vector clocks detect concurrent updates and enable proper conflict resolution.

YugabyteDB

PostgreSQL-compatible + cross-shard transactions. DocDB storage.