Path MTU Black Holes: Overlay Encapsulation Overhead, Filtered ICMP, MSS Clamping and Packetization-Layer Probing

The symptom is specific

A service behind a new tunnel, overlay network or VPN works for health checks and small API calls, then hangs on the requests that move a lot of data: a TLS handshake with a long certificate chain, a large JSON response, a file upload. The connection is established, nothing is refused, no error is logged, and after a long timeout the client gives up. The iptables documentation for the TCPMSS target lists the same pattern from the router's point of view: browsers connect and then hang, small mail works and large mail hangs, ssh works and scp hangs after the handshake.

The common cause is that a packet somewhere on the path is larger than a link can carry, the device that should report it cannot, and the sender keeps retransmitting the same oversized segment.

How path MTU discovery is supposed to work

RFC 1191 defines path MTU discovery for IPv4. The sender sets the Don't Fragment bit on its packets. A router that receives a packet larger than the next link's MTU drops it and returns an ICMP Destination Unreachable message with the code for fragmentation needed and DF set, carrying the next-hop MTU. The sender lowers its estimate for that destination and resends smaller packets. RFC 8201 does the same for IPv6, where routers never fragment at all, the message is ICMPv6 Packet Too Big, and every link must carry at least 1,280 bytes.

The mechanism depends entirely on the ICMP message getting back. Firewalls, security groups and cloud network policies that drop all ICMP, or that drop it because it arrives from a router address the sender never talked to, turn path MTU discovery into a black hole: the large packet disappears and the sender never learns why. RFC 4821 introduces its alternative as resolving the robustness problems of the classical technique precisely because it does not depend on the delivery of ICMP messages.

Where the bytes go in an overlay

Encapsulation adds headers to every packet, so the packet that the inner network sends must be smaller than the underlay MTU by the size of those headers. For VXLAN over IPv4, the format in RFC 7348 adds an outer IPv4 header of 20 bytes, a UDP header of 8 bytes and a VXLAN header of 8 bytes, and carries the inner Ethernet header of 14 bytes inside the payload. That is 50 bytes, so on a 1,500-byte underlay the inner interface must use 1,450. Over an IPv6 underlay the outer IP header is 40 bytes and the inner MTU drops to 1,430.

RFC 7348 also states that VXLAN tunnel endpoints must not fragment VXLAN packets, that intermediate routers may fragment the encapsulated packets, and that the destination endpoint may silently discard such fragments. An inner interface configured at 1,500 on a 1,500 underlay therefore produces exactly the silent loss described above. RFC 4459 discusses the same problem for tunnels in general and why none of the in-network fixes is free.

Confirming it

Test the path with packets of a known size and the DF bit set. On Linux, ping -M do -s 1472 sends a 1,500-byte IPv4 packet: 1,472 bytes of payload, 8 bytes of ICMP header and 20 bytes of IP header. If 1,472 fails and 1,422 succeeds across an overlay, the usable MTU is 1,450. tracepath reports the MTU it discovers hop by hop. On the sending host, ss -ti shows the PMTU and MSS each connection is using, and a packet capture shows the signature: full-sized segments retransmitted with growing back-off and never acknowledged, while small segments in the same connection are acknowledged normally.

Linux commands; the address is a placeholder for a host across the overlay.

ping -M do -s 1472 -c 3 10.20.0.15      # 1500-byte packets with DF set
ping -M do -s 1422 -c 3 10.20.0.15      # 1450-byte packets
tracepath -n 10.20.0.15
ss -ti dst 10.20.0.15 | grep -o "pmtu:[0-9]*\|mss:[0-9]*"

Four fixes and what each costs

The interface MTU is the fix to make first in any network you run. MSS clamping rewrites the maximum segment size option in TCP SYN packets so neither end ever sends a segment too large for the tunnel; the iptables TCPMSS target offers --clamp-mss-to-pmtu, which sets the MSS to the path MTU minus 40 for IPv4 or 60 for IPv6, and --set-mss for a fixed value. Its documentation warns that clamping to the path MTU may not work as intended with asymmetric routes.

Packetization-layer path MTU discovery, RFC 4821 for TCP and RFC 8899 for datagram protocols such as QUIC, does not rely on ICMP. The sender probes with larger packets and treats a lost probe as evidence about the MTU instead of congestion. Linux exposes it for TCP as net.ipv4.tcp_mtu_probing: 0 disables it, 1 enables it only when a black hole is detected, and 2 enables it always with an initial MSS of tcp_base_mss; tcp_probe_interval controls re-probing, every 10 minutes by default as in RFC 4821.

Worked example: the hang that only large responses hit

This scenario is illustrative. A team moves a service onto a cluster whose pods talk across a VXLAN overlay on a 1,500-byte underlay, and a configuration mistake leaves the pod interfaces at 1,500. Readiness probes pass, because their responses are a few hundred bytes. A client request that returns 40 kilobytes of JSON hangs. A capture on the sending pod shows 1,448-byte segments retransmitted repeatedly; on the underlay those segments became 1,550-byte packets, too large for the 1,500-byte links, and the fragmentation-needed messages from the underlay were dropped by a security group that allowed only TCP.

The quick mitigation was MSS clamping on the nodes, which made new connections work within minutes. The real fix was setting the overlay MTU to 1,450 in the network plugin configuration and rolling the nodes, after which the clamping rule was removed so that it could not hide the next misconfiguration. The security group was also changed to allow ICMP fragmentation-needed messages, so classic discovery works for traffic leaving the cluster.

Failure modes

• Blocking all ICMP as a security measure, which breaks path MTU discovery for every protocol on the path.
• Fixing TCP with MSS clamping and forgetting UDP: QUIC and DNS over large responses still fail unless they probe.
• Raising the underlay to jumbo frames on some hosts but not on every switch port in between, creating a new black hole.
• Assuming 1,500 bytes end to end through a VPN, a cloud interconnect or a provider with its own encapsulation; measure each path.
• Leaving a clamping workaround in place after fixing the MTU, which masks the next change that breaks it.

Practice and related reading

Further reading:

• TCP architecture in depth
• NAT in depth
• IPv6 in depth

Sources and scope

Checked against these sources on 20 October 2026. Examples and figures are illustrative, not measurements.

• RFC 1191: Path MTU Discovery
• RFC 8201: Path MTU Discovery for IP version 6
• RFC 4821: Packetization Layer Path MTU Discovery
• RFC 8899: Packetization Layer Path MTU Discovery for Datagram Transports
• RFC 7348: Virtual eXtensible Local Area Network (VXLAN)
• RFC 4459: MTU and Fragmentation Issues with In-the-Network Tunneling
• Linux kernel documentation: IP sysctl (tcp_mtu_probing, tcp_base_mss, tcp_probe_interval)
• iptables-extensions(8): TCPMSS target (--clamp-mss-to-pmtu, --set-mss)