Cilium 1.20.3: A Conntrack Garbage-Collection Race That Broke Live SNAT Connections, Network Policies That Enforced Nothing, and DNS Lost During Agent Restarts

Recent development · Practical technical guide · 21 October 2026

Cilium 1.20.3, released 15 October 2026, fixes a time-of-check to time-of-use race in batched conntrack garbage collection that could reallocate SNAT ports of live connections, marks CiliumNetworkPolicies with nodeSelector as invalid, and keeps DNS working through agent restarts.

Why a patch release deserves a read

Cilium 1.20.3 was published on 15 October 2026, the same day as 1.19.9 and 1.18.15. Most of its notes are routine backports, but three entries describe failures that are hard to diagnose from the outside: connections that reset or time out at a rate of a few per million, a network policy that reports nothing wrong while enforcing nothing, and pods that lose DNS whenever the agent restarts. Each one is worth understanding even if you do not run Cilium, because the underlying patterns, a stale snapshot acted on later, a validation that one component performs and another does not report, and a shutdown hook that never finishes its work, recur across infrastructure software. The release notes of 1.19.9 and 1.18.15 do not list these three fixes.

The conntrack race

Cilium keeps connection tracking (CT) entries in BPF maps, and a garbage collector in the agent deletes expired entries. Issue #46298 describes what went wrong after the collector switched to batched iteration: it reads a chunk of entries, decides which are expired, and deletes them afterwards. Between the read and the delete, the datapath can reopen the same five-tuple. For TCP, a SYN that matches a closing entry refreshes it in place. The collector, still holding its stale view, deletes an entry that is now live.

With BPF masquerading, deleting the CT entry also drops its NAT mapping. The next packet of the live connection gets a new SNAT source port and the server answers with a reset, or the reply misses reverse NAT and is dropped, which looks like a silent timeout. The reporter reproduced it with pod-to-world traffic at 7,500 connections per second from one node and saw about one failure per 400,000 connections, roughly 0.00025 percent, in a 24-hour soak; before batching, on 1.12, the rate was zero. The issue notes that a larger CT map makes the race easier to hit, because the read-to-delete window grows with the chunk processing time.

The fix, and what it does not claim

PR #46750, merged upstream on 23 September and backported to 1.20 in PR #49119, re-reads each entry immediately before deleting it and runs the expiry filter again on the current value. If the entry was refreshed, the delete is deferred to a later pass. The extra lookup runs only for entries already selected for deletion and uses a non-allocating lookup into a reused scratch value, so its cost is bounded.

The PR is explicit that this narrows the race rather than closing it: the datapath can still reopen an entry between the re-read and the delete, which restores roughly the window that existed before batching. In the author's soak test and production the failure rate dropped to zero. A compare-and-delete primitive in the kernel would close it fully; the PR says that approach was proposed upstream and is not being added, and that the long-term direction is in-kernel iterators.

Abridged from PR #46750 to show the shape of the change; see the PR for the exact code.

// Pattern from the fix: re-validate a stale decision against current state before acting
func (m *Map) purgeCtEntry(key CtKey, scratch *CtEntry, filter GCFilter, ...) error {
    if err := m.LookupInto(key, scratch); err != nil {
        return err
    }
    if m.filterEntry(filter, key, scratch) == noAction {
        return errDeferredReopened // refreshed since the batch read; reconsider later
    }
    return m.DeleteLocked(key)
}

Are you exposed

Condition	Why it matters
Cilium 1.19.4 or later on a branch without the fix	The issue reports the race from 1.19.4; among the 15 October releases only 1.20.3 lists the fix
BPF masquerading (SNAT) for pod-to-world traffic	Deleting the CT entry also drops the NAT mapping
High TCP connection churn with source-port reuse	HTTP/1.1 without keep-alive or clients that redial quickly
Large CT maps	A longer read-to-delete window

The symptom on the application side is a low, steady rate of connection resets or timeouts to external destinations that no retry policy fully hides and no single trace explains. If you see that on Cilium with BPF masquerading, this race is a candidate; check the masquerading mode in the cilium-config ConfigMap (enable-bpf-masquerade) and the agent version before chasing the remote side.

Policies that enforced nothing

A CiliumNetworkPolicy, the namespaced resource, may not use nodeSelector; only the clusterwide CiliumClusterwideNetworkPolicy may. PR #47882 explains the gap: the agent rejected such a policy and refused to enforce it, but the operator did not validate the field, so the policy's status did not show the error. A team could apply a host policy in a namespaced resource, see no error and believe it was in force. The v1.20 backport, PR #49173, makes the operator mark these policies invalid for both spec and specs. After upgrading, look for them before the new status surprises anyone:

Any result was never enforced; rewrite it as a CiliumClusterwideNetworkPolicy if a host policy was intended.

kubectl get cnp -A -o json | jq -r '.items[]
  | select(.spec.nodeSelector != null or ((.specs // []) | any(.nodeSelector != null)))
  | "\(.metadata.namespace)/\(.metadata.name)"'

DNS lost during agent restarts

Cilium can stop applying wildcard L7 DNS rules on shutdown so that pods under a toFQDNs policy keep resolving names while the agent and its DNS proxy are down. PR #48747 found that the shutdown path bumped the policy revision and regenerated endpoints without asking the policy computer to produce that revision, so each endpoint waited out a one-second timeout and failed, the DNS redirect stayed in the datapath, and pods lost DNS for the whole restart. The fix requests the recomputation explicitly and orders the shutdown so that the computation loop is still running. If agent upgrades have coincided with DNS failures in namespaces that use toFQDNs policies, this is the likely reason.

Rolling it out

Read the full 1.20.3 notes for your features; upgrade a canary node pool first.
Before upgrading, run the jq query above and fix or remove namespaced policies that use nodeSelector.
Measure the external connection reset rate per node before and after, with the same traffic, if you run BPF masquerading.
Restart agents on the canary pool deliberately and confirm that pods with toFQDNs policies keep resolving names.
On 1.19 or 1.18, watch those branches for backports of the CT fix rather than assuming it is included.

Patterns worth taking away

A batch read followed by later action on each item needs a re-check, or a primitive that makes the action conditional.
If one component rejects an object, the object status must say so; otherwise the rejection is invisible.
Shutdown hooks that depend on other components need explicit ordering and a test that runs them.

Sources and scope

Checked against these sources on 21 October 2026. Event: cilium/cilium v1.20.3, 15 October 2026; recheck it before acting. Examples and figures are illustrative, not measurements.

Upgrade to 1.20.3 where BPF masquerading meets high connection churn, find namespaced policies that use nodeSelector before upgrading, and test DNS through agent restarts.