All 79 articles, sorted alphabetically
Dapr 1.19.1 Security Fixes: Percent-Encoded Paths Past Access Control, Cross-App Reminders and What to Check
Dapr 1.19.1 and 1.18.10, released on 31 October 2026, fix a cluster of authorization bugs: deny rules bypassed with percent-encoded and semicolon paths, a token check skipped for an encoded health path, relayed gRPC calls, and workflow reminders delivered to the wrong app.
Read article →Metastable Failures: When Retries Keep a Service Overloaded After the Trigger Is Gone
Why a short slowdown can leave a healthy service down for good: the sustaining loop of timeouts, wasted work and retries, a simulation of when the system recovers and when it does not, and the changes that break the loop, from dropping expired work to retry budgets.
Read article →externalTrafficPolicy Local Versus Cluster: Client Source IPs, Uneven Load, Health Check Node Ports and Dropped Connections During Rollouts
What externalTrafficPolicy changes for LoadBalancer and NodePort Services: why Cluster hides the client IP behind an extra hop, how Local preserves it but can skew load and drop traffic, how the health check node port and terminating endpoints keep rollouts safe, and how to verify it.
Read article →Dapr 1.19: Stateful Workflow History, Cross-App Workflow Policies With requires, CRDs Before helm upgrade and Scheduler Placement
Dapr 1.19.0 shipped on 22 October 2026 with changes that affect how durable workflows are run and upgraded: workers that receive only new history, cross-app workflow operations gated by an access policy, a CRD step you must do before helm upgrade, and optional Scheduler-served actor placement.
Read article →Node Memory Pressure on Kubernetes: Kubelet Eviction Thresholds, the Kernel OOM Killer, QoS Scores and Which Pod Dies First
Two different mechanisms reclaim memory on a Kubernetes node, and they pick different victims. This explains kubelet node-pressure eviction versus the kernel OOM killer, how QoS class and requests decide who dies, and how to configure a node so the right pod goes first.
Read article →CPU Limits and CFS Throttling: Why a Pod at 40 Percent of Its Limit Still Stalls, and How to Measure and Fix It
How a Kubernetes CPU limit becomes a CFS quota per period, why multi-threaded bursts are throttled while average use stays far below the limit, how to read nr_throttled and throttled_usec, and how to choose between a larger limit, no limit, fewer threads or burst.
Read article →containerd 2.4.2, 2.3.7 and 2.2.10: Three Ways One Untrusted Image Can Take Down a Kubernetes Node, and the Admission Policy That Limits the Blast Radius
The containerd security releases of 13 October 2026 fix a daemon panic on a malformed manifest digest, a crash from a crafted Content-Length, and host disk exhaustion from an oversized layer stream: who is exposed and how to restrict image sources meanwhile.
Read article →Health Checks That Take a Fleet Down: Deep Readiness, Liveness Restarts, Load Balancer Fail-Open and Panic Thresholds
How a shared dependency outage plus deep health checks removes every instance at once, why liveness probes that check dependencies restart healthy processes, how ALB fail-open and Envoy panic thresholds differ from a Kubernetes Service with no ready endpoints, and what to check where.
Read article →PodDisruptionBudgets During Node Drains: Budget Arithmetic, Eviction 429s, Drains That Never Finish and unhealthyPodEvictionPolicy
How a PodDisruptionBudget decides whether kubectl drain may evict a pod: desiredHealthy and disruptionsAllowed, percentage rounding, 429 responses while replacements start, budgets that block forever, and what unhealthyPodEvictionPolicy changes for crash-looping pods.
Read article →Deadline Propagation Across Service Hops: Budgets That Shrink, Clock Skew, and the Work Nobody Will Wait For
How a request deadline should travel through a call tree: pass the remaining time rather than an absolute timestamp, check it before each stage, propagate cancellation when the request is doomed, and avoid the retry-past-deadline and bimodal-latency failures described in the SRE book.
Read article →Kubernetes Pod Termination Without Dropped Requests: EndpointSlices, preStop and the SIGTERM Race
Why rolling updates drop requests even with readiness probes: the race between endpoint propagation and SIGTERM, how preStop sleep, listener draining, keep-alive handling and the grace period fit together, and how to prove zero errors with a load test.
Read article →The CAP Theorem
The CAP Theorem explains why distributed systems cannot guarantee consistency, availability, and partition tolerance simultaneously. Understand the tr…
Read article →In Search of a Leader: Understanding Raft Consensus - AICassindra
In the world of distributed systems, getting a cluster of nodes to agree on something—like the order of log entries—is notoriously difficult.
Read article →API Gateway Patterns
Kong, Ambassador, AWS API Gateway. Auth (JWT validation), rate limits (per API key), routing (path-based, host-based, weighted), request/response tran…
Read article →Bulkhead Pattern
Isolate resources per feature. Prevent noisy neighbor.
Read article →Byzantine Fault Tolerance, in depth: the 3f+1 bound, PBFT's three phases, HotStuff's linear view change, and when BFT earns its cost
Byzantine fault tolerance from first principles: what the Byzantine fault model covers, why n must be at least 3f+1, partial synchrony and what it mea…
Read article →CAP Theorem + PACELC Extension, in depth: what the proof actually says, the latency cost of consistency when nothing is broken, and choosing per operation
CAP and its PACELC extension explained precisely: the Gilbert and Lynch definitions, three common misreadings, why replication forces a latency versus…
Read article →Chaos Engineering, in depth: injecting partitions, pauses, latency and clock faults into a distributed system and checking safety, not just uptime
Chaos engineering for distributed systems: a partial-fault model, tc netem, iptables, Toxiproxy and Chaos Mesh injection by layer, safety hypotheses c…
Read article →Circuit Breaker Pattern
Fail fast when dependency down. Prevent cascade failures.
Read article →Clocks in Distributed Systems, in depth: physical drift, wall versus monotonic time, timestamp ordering, Lamport clocks and choosing the right clock
A map of time in distributed systems: how hardware clocks drift and get synchronized, wall-clock versus monotonic clocks in code, why last-write-wins …
Read article →Consistent Hashing, in depth: Rings, Virtual Nodes and the Alternatives
Consistent hashing from first principles: ring lookup code, why one token per node is unbalanced and what virtual nodes cost, measured key movement, z…
Read article →CQRS, in depth
CQRS without event sourcing: the three levels of separation, command handlers with optimistic versions, why dual writes fail, outbox and CDC propagati…
Read article →CRDTs in Production, in depth: choosing data types, syncing with state vectors, storing and compacting history, evolving schemas and securing merges
What it takes to run conflict-free replicated data types in a real product: picking counter, register, set, map and sequence types by their surprising…
Read article →Anti-Entropy Repair: Merkle Trees, Read Repair, Hinted Handoff
How anti-entropy repair heals replica divergence: Merkle tree diffs, read repair, hinted handoff, background repair and how Dynamo, Cassandra and Riak…
Read article →Bounded Staleness Consistency Explained: Lag Bounds and Reads
Bounded staleness consistency explained: bounds of k versions or t seconds, replica lag, serving or redirecting reads, and session tokens for read-you…
Read article →Causal consistency architecture
Deep-dive on causal consistency: the happened-before relation and session guarantees, dependency tracking with vector clocks, hold-until-ready applica…
Read article →Chain Replication Explained: Head-to-Tail Writes, CRAQ Reads
How chain replication gives linearizable consistency: writes flow head to tail and commit at the tail, tail reads, reconfiguration on failure, and CRA…
Read article →Google Chubby, in depth: a coarse-grained lock service, its sessions, KeepAlives and cache invalidation, master failover, and the lessons that shaped ZooKeeper and etcd
How Google's Chubby lock service works, from the 2006 OSDI paper: why a lock service rather than a consensus library, the cell and master lease, …
Read article →Raft consensus architecture, in depth: deploying a Raft group, commit latency maths, placement, Multi-Raft and operations
Raft as a deployed component: where consensus belongs, the node pipeline from proposal to apply, commit latency from fsync and placement, sizing and m…
Read article →Raft Consensus, in depth: from the rules in the paper to a production node
Raft as you would implement and operate it: persistent state and fsync ordering, the vote and append handlers as code, why leaders only count replicas…
Read article →CRDT replication architecture
Deep-dive on CRDT replication: state-based vs op-based vs delta CRDTs, join-semilattice merges, OR-Sets and PN-counters, version vectors and dots, seq…
Read article →Distributed Hash Table Architecture in Depth
DHT architecture in depth: one id space for keys and nodes, virtual nodes, Chord fingers vs Kademlia k-buckets vs Pastry prefixes, join/leave converge…
Read article →etcd, in depth: the revisioned key space, Raft, transactions, watches and leases, and how to keep a cluster healthy
etcd from first principles: what it is for, how a member is built (gRPC, Raft, WAL, MVCC on bbolt), revisions and history, linearizable versus seriali…
Read article →What Is a Fencing Token? Safe Distributed Locks Despite Pauses
Fencing tokens explained: why a GC-paused lock holder breaks lease-based locks, a monotonic token per grant, stale tokens rejected, ZooKeeper zxid.
Read article →Gossip-Based Membership, in depth: SWIM probing, suspicion and incarnations, piggybacked dissemination and Lifeguard
How clusters agree on who is alive without a coordinator: why all-to-all heartbeats do not scale, the SWIM probe cycle with indirect pings, suspicion …
Read article →Gray failure architecture
Deep-dive on gray failure: the degraded component whose shallow health check passes while real requests suffer, why differential observability is both…
Read article →Hinted Handoff Explained: Hints, Hint Window, Replay and Repair
How hinted handoff keeps a Dynamo-style store writable when a replica is down: a live node stores a hint, replays it later, and repair covers lost hin…
Read article →Hybrid Logical Clock (HLC): Physical Time Plus Causality
Hybrid logical clocks (HLC) explained: the update rules, why the clock stays near physical time, the 64-bit encoding, bounded skew, and use in Cockroa…
Read article →Distributed Leader Election Architecture in Depth
Leader election as a two-layer architecture: Raft elections inside etcd, PreVote and CheckQuorum, lease-based election for your own service, Kubernete…
Read article →Leases -- time-bounded exclusive rights
Deep-dive on distributed leases: the safe-exclusive-access need, the time-bounded grant, expiry (auto-release), renewal (by the live holder), contrast…
Read article →Merkle tree architecture
Deep-dive on Merkle trees for replica reconciliation: leaf hashes over key ranges, parent hashes up to a single root, root-then-descend comparison, dr…
Read article →Operational transformation architecture
Deep-dive on OT for collaborative editing: the operation model, the transformation function and TP1/TP2, the Jupiter central-server model, client pend…
Read article →Paxos Consensus Algorithm Explained: Phases, Quorums, Multi-Paxos
How Paxos reaches consensus: proposers, acceptors and learners, prepare and accept phases, 2F+1 quorums, ballots, Multi-Paxos with one prepare, and vs…
Read article →Phi-accrual failure detectors, in depth: per-peer state, Cassandra and Akka internals, pause guards and threshold calibration
How phi-accrual failure detectors work inside production systems: per-peer windows, the exponential model in Cassandra and the normal model in Akka, l…
Read article →Quorum systems
Deep-dive on quorum-based replication: the R+W>N overlap guarantee, tunable consistency levels, timestamp vs vector-clock versioning, read repair a…
Read article →Raft Log Replication: AppendEntries, Majority Commit
How Raft replicates its log: AppendEntries from the leader, majority commit, the log-matching invariant, conflict overwrite, nextIndex/matchIndex, sna…
Read article →Read repair and anti-entropy architecture
Deep-dive on the repair mechanisms behind leaderless eventual consistency: read repair on the coordinator's read path with the digest optimizatio…
Read article →Saga Pattern, in depth: compensations, the pivot step, a durable orchestrator and the isolation you give up
How the saga pattern keeps multi-service business transactions consistent without distributed locks: compensatable, pivot and retriable steps, orchest…
Read article →Split Brain, in depth: how two leaders happen, what they break, and how quorum, leases, fencing and STONITH stop them
Split brain explained from first principles: how network partitions, asymmetric links and process pauses create two active leaders, what dual writes d…
Read article →Total Order Broadcast: Atomic Broadcast and Consensus
Total order broadcast: agreement, total order, validity and integrity, why it is equivalent to consensus, leader-sequenced logs, state-machine replica…
Read article →Vector clock architecture, in depth: causal delivery, consistent cuts and debugging distributed traces
Vector clocks as event-level infrastructure: a clock layer between application and transport, causal broadcast with a hold-back queue, recovering cons…
Read article →Vector Clocks in Depth: Tracking Causality, Detecting Concurrent Writes and Keeping the Metadata Bounded
How vector clocks capture happens-before exactly: first principles, the update and comparison rules with Python code, a worked three-process example, …
Read article →Replication watermarks architecture
Deep-dive on watermarks in a replicated log: the high-water mark as the committed, safe-to-read boundary computed from the minimum offset replicated t…
Read article →ZooKeeper architecture, in depth: znodes, the ensemble, Zab, sessions, watches, consistency guarantees and how to run it
How Apache ZooKeeper works from the inside: the znode data model and node types, leaders, followers and observers, the Zab atomic broadcast and zxids,…
Read article →Distributed Locking Patterns, in depth: database locks, Redis, Redlock, ZooKeeper and etcd, and fencing
A catalogue of distributed locking patterns and how to choose: efficiency versus correctness locks, why leases need fencing tokens, database row and a…
Read article →Distributed Tracing with OpenTelemetry, in depth: causality, the propagation contract, trust boundaries, clock skew and trace completeness
Distributed tracing as a distributed-systems problem: the causal span model, W3C traceparent, tracestate and baggage, where propagation breaks, durabl…
Read article →Distributed Transactions, in depth: 2PC and 3PC as state machines, termination rules and a crash-point model check
Atomic commitment from first principles: 2PC and 3PC as participant state machines, why the prepared state blocks, how the 3PC pre-commit round and te…
Read article →Calvin, in depth: deterministic distributed transactions without two-phase commit
How Calvin agrees on transaction inputs before execution: 10 ms sequencer epochs, Paxos or async input replication, deterministic locking, active and …
Read article →Distributed Transactions in Practice, in depth: choosing between co-location, XA, distributed SQL, sagas and idempotent retries
A practical guide to making changes atomic across machines and services: the decision ladder, how two-phase commit and XA fail in production, how Perc…
Read article →The Dynamo Paper, in depth: how Amazon's always-writable key-value store composed six techniques, and what its descendants kept
A first-principles reading of Amazon's 2007 Dynamo paper as a whole system: the requirements that forced each design choice, how consistent hashi…
Read article →Event Streaming Patterns, in depth: notification vs state transfer, fan-out, outbox, compacted tables, idempotent projections and dead letters
A practical catalogue of event streaming patterns on a partitioned log: event notification versus state transfer, fan-out with consumer groups, outbox…
Read article →Failure Detectors and Phi Accrual, in depth: completeness, accuracy, the detector classes, and building timeout, phi and SWIM detectors that behave in production
How distributed failure detectors really work: why they can only suspect, completeness and accuracy, the Chandra-Toueg classes, detection time and mis…
Read article →Google Spanner and TrueTime, in depth: commit wait and its proof, timestamp rules, safe time for replica reads and what to copy without atomic clocks
How Spanner uses TrueTime for external consistency: the interval API and the 2012 uncertainty figures, the Start and Commit Wait rules with the four-s…
Read article →Gossip Protocol for Cluster Membership, in depth: epidemic dissemination, versioned state and digest reconciliation
How gossip spreads cluster membership state: push, pull and push-pull epidemics with a simulation and measured round counts, versioned endpoint state …
Read article →What Jepsen Taught Us, in depth: histories, nemeses, checkers and the failure patterns that keep coming back
What a decade of Jepsen analyses teaches about distributed systems: how a Jepsen test is built, why indeterminate operations matter, how Knossos and E…
Read article →Kubernetes Operator Pattern, in depth: level-triggered reconcile loops, CRD design, finalizers and running operators safely
How Kubernetes operators work from first principles: informers, deduplicating workqueues and level-triggered reconciliation, spec and status design wi…
Read article →Leader election algorithms, in depth: bully, ring, quorum-and-term and lease-based election compared, with code and fencing
A comparative guide to leader election algorithms: the safety and liveness problem, the bully algorithm, ring algorithms (LCR, Chang-Roberts, Hirschbe…
Read article →Linearizability vs Sequential Consistency, in depth: histories, real-time order, a working checker, composability, cost, and where real systems land
A first-principles comparison of linearizability and sequential consistency: operation histories, the one difference (real-time order), worked histori…
Read article →Quorum Systems Explained, in depth: the intersection property, majorities, weighted votes, grids, Flexible Paxos and Byzantine quorums
Quorum systems from first principles: why intersection is the only property that matters, majority and weighted voting, grid quorums and the load vers…
Read article →Quorum Writes and Reads, in depth: the coordinator request path, what a timeout really means, and why R plus W over N is not linearizable
What happens on a quorum write and read in a Dynamo-style store: the coordinator, digest reads, blocking read repair, hinted handoff and sloppy quorum…
Read article →Raft Consensus Algorithm, in depth: the five safety properties as code, deterministic simulation, and the bugs real implementations shipped
How to know a Raft implementation is correct: the algorithm in brief, the paper's five safety properties written as an executable checker, a dete…
Read article →Raft Consensus Intuition, in depth: deriving every rule by breaking a simpler design
Build Raft from first principles: start with a primary and a backup, break it with partitions and crashes, and add one rule per failure until you reac…
Read article →Retry, Exponential Backoff and Jitter, in depth: what to retry, jitter variants, retry budgets, deadlines and retry amplification
How to retry safely in distributed systems: classifying retryable errors, idempotency keys, exponential backoff with full, equal and decorrelated jitt…
Read article →Saga: Choreography vs Orchestration, in depth
Saga choreography versus orchestration in depth: one checkout saga built both ways in code, the cost of adding a step, how each style detects a stuck …
Read article →Istio vs Linkerd, in depth: sidecar, ambient and micro-proxy data planes, identity, policy, upgrades and choosing a mesh
Choosing and operating a service mesh: Istio sidecar and ambient mode (ztunnel, HBONE, waypoints) versus Linkerd's Rust micro-proxy, control plan…
Read article →Sharding Strategies, in depth: choosing a shard key from your workload
How to choose a sharding scheme from evidence: partition keys versus partition functions, the balls-into-bins imbalance bound and heavy-tailed keys, a…
Read article →Two-Phase Commit, in depth: building a coordinator, recovering in-doubt branches and testing every crash point
A hands-on guide to two-phase commit: a runnable coordinator over PostgreSQL prepared transactions, a fenced recovery loop, a crash-point test matrix,…
Read article →Two-Phase Commit Revisited, in depth
Two-phase commit at the level of log records: the basic protocol, presumed abort and presumed commit, read-only and one-phase optimisations, the block…
Read article →YugabyteDB, in depth: DocDB tablets, Raft per shard, hybrid time transactions and multi-region design
How YugabyteDB works: the YSQL query layer over DocDB, hash and range sharding into tablets, Raft replication with leader leases, hybrid time and dist…
Read article →