Why architecture matters here
Architecture matters here because conntrack silently converts a rate into a level. Your application thinks in connections per second; conntrack thinks in concurrent entries, and the conversion factor is the timeout. Ten thousand short-lived connections per second with a 120-second TIME_WAIT timeout is not ten thousand entries — it is over a million. Capacity is rate multiplied by lifetime, and the lifetime is usually a default nobody chose.
The failure is also positioned to be maximally damaging. Conntrack lives on gateways, NAT boxes, Kubernetes nodes, and load balancers — exactly the shared chokepoints whose failure is not one service degrading but every service behind them failing at once. And because the table is per-host, the blast radius follows the topology, not the application.
Finally, the behaviour at the limit is the opposite of what most caches condition you to expect. A full conntrack table does not evict the oldest entry to admit the newest. It drops the new packet and logs 'nf_conntrack: table full, dropping packet'. The system protects the flows it already knows and refuses everything else, which is why the symptom is so cleanly split between old and new connections.
That cliff shape is the last reason architecture matters here, and it is the cruellest one. Most saturating resources warn you: a filling disk slows down, a loaded CPU raises latency, a hot cache lowers its hit rate. Conntrack does none of this. At 99 percent utilisation it performs exactly as well as at 10 percent, and at 100 percent it refuses every new connection instantly. There is no degradation curve to notice, no gradual latency creep to alert on — which is precisely why the one gauge of count over max is worth more than any amount of downstream symptom monitoring. It is the only signal that exists before the cliff.
The architecture: every piece explained
The tuple and the entry. Conntrack identifies a flow by a tuple — source address, destination address, source port, destination port, and protocol — in both directions. Each tracked flow gets an nf_conn struct holding the original tuple, the expected reply tuple, the current state, a timeout, and any NAT bindings. The reply tuple is what makes return traffic matchable without a separate rule.
The hash table. Entries live in a hash table sized by nf_conntrack_buckets (the hashsize), with collisions chained. The total entry ceiling is nf_conntrack_max, a separate number. The ratio matters: too few buckets for the entry count turns lookups into linked-list walks, so the conventional guidance is to keep max at roughly four times hashsize, and to raise both together rather than only raising max.
The state machine. Conntrack states are not TCP states, though they borrow the vocabulary. A flow starts NEW, becomes ESTABLISHED once traffic is seen in both directions, and passes through closing states to TIME_WAIT. Crucially conntrack tracks UDP and ICMP with the same machinery despite neither having connections — it synthesises a notion of established from bidirectional traffic and a timer.
Timeouts are the capacity knob. Each state has its own timeout in /proc/sys/net/netfilter/: tcp_timeout_established defaults to five days, tcp_timeout_time_wait to 120 seconds, udp_timeout to 30 seconds. The five-day established default is the single most misunderstood number in the subsystem — it exists so that idle SSH sessions survive, and it means a leaked connection occupies a slot for the better part of a week.
Where entries actually live. Each nf_conn costs a few hundred bytes of unswappable kernel slab, so the table is a real memory commitment: a max of four million is roughly a gigabyte you can never page out. Conntrack is also per-network-namespace, which is the detail that surprises container operators — the host, and potentially each namespace, keeps its own table and its own counters. On a Kubernetes node the table that matters is usually the host's, seen through whichever namespace your monitoring agent happens to run in, and reading the wrong one produces a reassuring graph of an irrelevant table.
NAT on top. When NAT is in play, conntrack holds the binding, and the port space becomes a second, tighter ceiling. A single NAT source address has roughly 64k ports per destination tuple; masquerading thousands of clients behind one address exhausts ports long before the table fills, producing the same 'new connections fail' symptom from an entirely different resource. Diagnosing one while the other is the constraint is a classic wrong turn.
The escape hatch. Not every packet needs tracking, and the raw table's NOTRACK target says so explicitly: matching flows skip conntrack entirely, consuming no entry and no lookup. This is the right tool for traffic that needs neither NAT nor stateful filtering — bulk health checks, high-volume UDP telemetry, known internal flows. The trade is absolute and worth stating plainly: a NOTRACK flow gets no stateful firewall protection and no NAT, because there is nothing remembering it. You are buying capacity with state, which is exactly the right trade for flows whose state you were never going to consult.
End-to-end flow
Trace a packet. A client behind a NAT gateway opens a TCP connection. The SYN arrives at the gateway's PREROUTING hook, where conntrack computes the tuple hash and looks it up. No entry exists, so this is a NEW flow: conntrack allocates an nf_conn, fills the original tuple, and computes the expected reply tuple.
The packet continues to the NAT rules, which pick a source port and rewrite the address. That binding is stored in the same entry, so the reply tuple now reflects the post-NAT view. This is why NAT without conntrack is impossible: the reverse translation is not derivable from the return packet alone — it must be remembered.
The SYN-ACK returns. Its tuple matches the stored reply tuple, so conntrack finds the entry immediately, marks the flow ESTABLISHED, and applies the reverse NAT rewrite. The firewall rule that allows this packet is not a rule about port 443 at all — it is -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT, which is stateful precisely because the entry exists.
Data flows. Each packet refreshes the entry's timer to the established timeout — five days by default. Note what this means: the timer is refreshed, not counted down from creation, so an active connection never ages out and an abandoned one waits the full window. That refresh semantics is why a leak is so durable: an entry whose peer vanished without a FIN or RST has nothing left to refresh it, but it also has five days of runway before the kernel reclaims it, and TCP keepalives are the only mechanism that will notice the peer is gone any sooner.
The client closes. Conntrack walks the closing states and lands in TIME_WAIT with its 120-second timer. The flow is over from the application's perspective, but the entry is very much alive. At ten thousand connections per second, that 120-second tail alone is 1.2 million resident entries — comfortably above a default nf_conntrack_max of 262,144.
Now the ceiling. A new SYN arrives, conntrack tries to allocate, the table is full. There is no LRU eviction. The kernel drops the packet and rate-limits a 'table full' message to dmesg. The client sees a SYN timeout and retries; the retries are also dropped. Every established flow keeps working perfectly, which is exactly why the on-call engineer spends forty minutes looking at the application before someone checks nf_conntrack_count.
Watch how it resolves, because the recovery shape is as instructive as the failure. Nothing the application team does helps: restarting pods makes it worse, since each restart abandons established entries that keep their timers and adds new NEW packets that get dropped. The table drains only as timers expire, on conntrack's schedule, not yours. Raising nf_conntrack_max at runtime works instantly and buys room, but if hashsize stays put you have traded the cliff for lengthening hash chains and rising softirq CPU. The durable fix is upstream of all of it — shorten tcp_timeout_time_wait so the 120-second tail stops multiplying your connection rate into a resident level you never intended to hold.