Why architecture matters here
Gossip fails on flapping (falsely marked dead + alive), on network partitions (split membership), and on bandwidth blowout (fanout too high). Architecture matters because tuning fanout + suspicion + anti-entropy shapes the whole cluster's health.
The architecture: every piece explained
The top strip is the pattern. Peer selection picks random subset. Rumor spreading shares state deltas. Ack + suspicion detects failures — no ack means suspect, then dead. Indirect ping asks intermediaries before declaring a peer dead — reduces flapping.
The middle row is optimization. Anti-entropy reconciles divergent state. Piggyback carries multiple updates per gossip. Convergence time is O(log N) rounds. Bandwidth cap tuned via fanout.
The lower rows are practice. Membership list tracks cluster members. Observability watches flapping + convergence. Ops covers seeding + partition handling + drills.
End-to-end flow
End-to-end: node A gossips to random 3 peers every second. Peer B is unresponsive. A gossips indirect ping via C + D; both fail; A marks B suspect. If B doesn't respond in suspicion window, A declares dead. Gossip propagates. Cluster converges in O(log N) rounds. Anti-entropy runs periodic full reconciliation.