Why architecture matters here
The economics of a datacenter network are the economics of parallelism. You cannot buy a single link fast enough to carry a rack's aggregate egress, and even if you could, you would have no redundancy. So fabrics are built wide: a leaf switch connects upward to many spines, and between any two servers in different racks there are dozens or hundreds of equal-length paths. That topology is only worth its cost if traffic actually uses all those paths. ECMP is the mechanism that turns latent parallel capacity into delivered bandwidth, and it is why a well-built fabric can offer full bisection bandwidth — any server able to talk to any other at line rate — instead of a fabric where half the links sit dark.
The architectural constraint that shapes everything is flow affinity. TCP's fast-retransmit and congestion control interpret out-of-order delivery as loss; scatter a single connection's packets across four paths with different queuing delays and you manufacture spurious retransmits, collapse the congestion window, and tank throughput. Stateful middleboxes (NATs, firewalls, load balancers doing connection tracking) are worse: they must see every packet of a connection to maintain state, so a flow that wanders between paths that traverse different middleboxes simply breaks. ECMP's per-flow hashing exists to honor this constraint: it is a load-spreading scheme deliberately restrained to never split a flow.
That restraint is also ECMP's central limitation, and understanding it is the whole game operationally. Because the unit of spreading is the flow, ECMP balances flows, not bytes. If your traffic is a million small flows, the law of large numbers gives you near-perfect balance. If it is a handful of enormous flows — a backup replication stream, a Spark shuffle, a storage rebuild — a single hash bucket can pin 40Gbps of elephant onto one link while its neighbors idle, and no amount of extra links helps because the flow cannot be split. Every advanced ECMP feature — flowlet switching, dynamic load balancing, weighted ECMP — is a response to this one uncomfortable fact.
It is worth being precise about why the stateless design is a feature rather than a limitation to be engineered away. A stateful load balancer that tracked every flow could balance bytes perfectly, but it would need a per-flow table sized to the peak concurrent flow count — millions of entries at a busy switch — synchronized across the redundant control planes, and consulted on every single packet at terabit line rate. That table becomes a scaling bottleneck, a failure domain, and a synchronization headache. ECMP's hash trades that away entirely: the 'decision' for a flow is recomputed from the packet header on every packet and always yields the same answer, so there is nothing to store, nothing to replicate, and nothing to lose when a linecard reboots. The determinism of the hash is the state, encoded in the algorithm rather than in memory. This is the same insight that makes consistent hashing valuable in distributed systems generally, applied at the hardware forwarding plane where the packet rate makes any per-flow bookkeeping economically impossible.
The architecture: every piece explained
Top row: the decision path. The routing table is where ECMP begins — when the routing protocol (BGP in datacenters via BGP-in-the-fabric, or an IGP like OSPF/IS-IS) installs multiple routes to the same prefix with identical metric, the FIB collapses them into a single entry pointing at a next-hop group of N members. The hash function reads a configured set of header fields — the canonical 5-tuple, sometimes extended with inner headers for tunneled traffic (VXLAN, GRE) so encapsulated flows still spread — and produces an index. Crucially the hash must be symmetric-agnostic but deterministic: the same packet always yields the same member, computed in the ASIC with no state lookup. The granularity choice — per-flow (one path for the connection's life) versus flowlet (re-decide during natural gaps in a flow) — trades perfect ordering safety against better balance.
Middle row: the members. Each link is one next-hop — a physical interface toward a specific spine, or in server load balancing, one backend in the pool. The hash index modulo the group size selects the member. The subtlety is what happens when the group changes size: a naive modulo remaps almost every flow when you add or remove a member (hash % 4 versus hash % 5 agree on almost nothing), which is catastrophic churn. Production implementations use resilient hashing (a consistent-hash-like member table) so adding or draining one link disturbs only the flows that were on the changed member, not the whole population.
Bottom rows: the fabric context. In a Clos spine-leaf topology, ECMP is applied at every hop: the leaf hashes to choose a spine, and because all spines are equidistant, the flow can take any of them — this is how bisection bandwidth is realized. Rehash on member change with resilient hashing keeps a spine failure from reordering surviving flows. The ops strip is the reality of running this: watching for hash polarization (where successive hops make correlated choices and traffic funnels onto a subset of links), monitoring per-link utilization and flow entropy, and using drain/undrain to remove a link for maintenance gracefully rather than yanking it.
End-to-end flow
Trace a packet from a web server in rack 7 to a database in rack 22 of a two-tier Clos fabric with 32 spines. The server's leaf switch has a route to the database's subnet learned from all 32 spine uplinks with equal cost, so the FIB entry points at a 32-member next-hop group. A new TCP connection opens: SYN packet arrives, the ASIC hashes its 5-tuple — source port was chosen randomly by the client stack, which is exactly the entropy ECMP relies on — and the hash lands on spine 11. The SYN goes to spine 11, which itself has equal-cost routes down to rack 22's leaf (it connects to every leaf), hashes the same 5-tuple, and forwards down. Every subsequent packet of this connection — data, ACKs, the eventual FIN — hashes identically and rides the exact same leaf → spine 11 → leaf path. TCP sees in-order delivery; the connection never knows it is sharing the fabric with hundreds of thousands of siblings.
Now multiply: 200,000 concurrent connections from rack 7, each with an independent random source port, hash across the 32 spines. Because the source ports are effectively uniform, roughly 6,250 flows land on each spine, and aggregate load is even to within a few percent. The fabric delivers its full cross-sectional bandwidth, and no operator ever thinks about it. This is ECMP working as designed: statistically flat, per-flow consistent, stateless.
The interesting case is the stress path. At 2am a storage rebuild kicks off: three 25Gbps replication flows between two racks. Three flows, three hash buckets — and bad luck puts two of them on spine 11. That link is now carrying 50Gbps of elephant on top of its background mice, hits congestion, and starts dropping. The other 31 spines are fine. Per-flow ECMP cannot help — splitting a replication flow would reorder it. If the fabric supports flowlet switching, the story changes: during the micro-bursts and gaps inherent even in a bulk transfer, the switch detects an inter-packet gap large enough that no in-flight packets remain on the old path, and re-hashes the flowlet onto a less-loaded spine — moving load without reordering, because the gap guaranteed the pipe was empty. Finally, maintenance: an operator drains spine 11. With resilient hashing, only the flows currently pinned to spine 11 rehash onto survivors; the other 31/32 of traffic never moves, so the drain is nearly invisible instead of a fabric-wide reordering event.
Zoom out to the fabric-wide view of that same busy afternoon and a second-order effect appears that pure per-flow reasoning misses: hash polarization. Imagine every switch in the two tiers ships with the same default hash seed. The leaf hashes a flow and picks spine 11; spine 11 hashes the same five-tuple with the same algorithm and seed, so its choice of downstream link is correlated with the leaf's, not independent. Across thousands of flows, these correlated decisions cause traffic to funnel onto a subset of the theoretically available paths — the fabric looks balanced at the first hop but bunches at the second, and some links run hot while others sit at ten percent. The operator sees unexplained congestion despite 'plenty of capacity', and the fix is not more links but decorrelating the hops: give each switch a distinct per-device seed so the second hop's choice is statistically independent of the first. This is the kind of failure that only surfaces at fabric scale, never in a two-switch test, which is exactly why ECMP deployments are validated with per-link utilization heatmaps rather than a presumption of uniformity.