Why architecture matters here
The architecture matters because the failure it prevents is uniquely deceptive. A PMTU problem does not look like a network problem — the connection establishes perfectly, because the handshake packets are small and cross any path. It only breaks when the application sends something large: a big HTTP response, a file upload, a database result set. So the symptom is 'small requests work, big ones hang,' which points suspicion at the application, the server, or the payload, when the real cause is a link three hops away silently dropping oversized packets. An engineer can burn hours in the wrong layer because the failure mode is invisible to every tool that only sends small probes.
It matters because it is fundamentally a per-path, changeable quantity, not a fixed setting. There is no single MTU you can configure once; the path MTU to one destination may be 1500 and to another 1400, and either can change when routing shifts a flow onto a different link with a different MTU. That is why the value must be discovered dynamically and cached per destination, and why the cache must age out and re-probe — a PMTU learned an hour ago may be stale after a reroute. An architecture that treats MTU as a static host setting will be right on the paths it was tuned for and silently wrong on the rest, so the discovery-and-cache design is what makes correct behavior portable across a heterogeneous network.
It matters because the trade-offs of the alternatives are all bad, which is why discovery is worth the trouble. You could fragment every oversized packet, but fragmentation is inefficient, hurts performance, is a reassembly and security headache, and is simply not available for IPv6 routers or DF-set IPv4. You could set every packet conservatively small (a low MSS everywhere), but then you pay per-packet overhead on every transfer across every path, wasting bandwidth on the many links that could have carried full-size frames. PMTUD exists precisely to get the best of both — full-size packets where the path allows, smaller only where it must — and the cost of that optimization is the discovery machinery and its dependence on ICMP, which is the fragility the rest of the architecture works around.
Finally, it matters because tunnels and overlays have made sub-1500 path MTUs the common case, not the exception. VPNs, GRE, IPsec, VXLAN, and cloud overlay networks all wrap packets in extra headers, shrinking the usable MTU below the 1500 that endpoints assume. A modern data-center or cloud path frequently has an effective MTU of 1400-ish somewhere in the middle, so PMTUD (or a clamp that stands in for it) is not an edge case but a routine requirement. This is why MSS clamping at tunnel boundaries — rewriting the TCP handshake's maximum segment size to fit the tunnel — has become standard practice: it makes the endpoints agree to a safe size up front so the black hole never forms.
The architecture: every piece explained
Top row: discovery working as designed. The sender starts with its local link MTU (1500 on Ethernet) as its guess and sends full-size packets with the don't-fragment bit set. Somewhere along the path is a link with a smaller MTU — say a 1400-byte tunnel. That link cannot forward the 1500-byte packet without fragmenting, and DF forbids fragmenting, so instead of dropping silently it returns an ICMP 'fragmentation needed' message (type 3, code 4 in IPv4; 'packet too big' in ICMPv6) that carries the next-hop MTU, 1400. Receiving it, the sender lowers its path MTU for that destination from 1500 to 1400 and resends the data in 1400-byte packets that now cross the whole path. Discovery has converged in one round.
Middle-left: the failure the architecture must survive. If the ICMP is filtered — a firewall blocking ICMP, a load balancer not relaying it, an asymmetric route — the 'fragmentation needed' message never reaches the sender. This is the blackhole: the router still drops the oversized packet (it must, with DF set), but the sender never learns why. So big packets vanish while small ones pass, and because the handshake used small packets, the connection came up fine and then hangs the instant it sends bulk data. The sender keeps retransmitting the same too-big packets, each silently dropped, until the connection times out.
Middle-right: the two robustness mechanisms that do not depend on ICMP. PLPMTUD (packetization-layer PMTUD) has the transport itself probe: it sends progressively larger packets and infers the path MTU from which sizes get acknowledged versus lost, so it discovers the right size without any ICMP message — immune to the blackhole. MSS clamping takes a different angle: a device at the tunnel edge rewrites the maximum-segment-size option in the TCP handshake down to what the tunnel can carry, so both endpoints agree from the start to send segments that fit and never emit an oversized packet at all. One recovers from the blackhole; the other prevents it.
Bottom row and ops: caching and the operational surface. The sender caches the discovered PMTU per destination, reuses it for subsequent packets, ages it out after a while, and periodically re-probes upward in case the path changed to allow larger packets again — so discovery is amortized, not repeated per packet, yet still adapts. When it all works the path is stable at the right size: no fragmentation, no blackhole, full-size packets wherever the path allows and smaller only where it must. The ops strip names the practical rules: allow ICMP type 3 code 4 (never blanket-block ICMP), clamp MSS on tunnels so sub-1500 overlay paths are safe by construction, keep a PLPMTUD fallback for paths where ICMP is unreliable, and watch for stalled large transfers as the fingerprint of a blackhole.
End-to-end flow
Trace a single HTTPS download crossing a VPN tunnel with a 1400-byte MTU, first in a healthy network and then through a firewall that blocks ICMP, to see discovery succeed and then collapse — and to see the clamp save it.
The handshake succeeds either way: the client opens the connection. The TCP SYN, the TLS ClientHello, and the early handshake packets are all small — well under 1400 bytes — so they cross the 1400-byte tunnel with room to spare regardless of MTU discovery. The connection establishes cleanly. This is the trap: everything looks healthy, because nothing large has been sent yet.
Healthy path — discovery converges: the server begins sending the response body in full 1500-byte segments with DF set. The first one hits the tunnel ingress, which cannot forward 1500 bytes and returns an ICMP 'fragmentation needed' carrying MTU 1400. The server receives it, lowers its cached path MTU for this client to 1400, and retransmits the data in 1400-byte segments. From here the download streams at full speed; the server reuses the cached 1400 for the rest of the transfer. One brief round of discovery, then steady state — exactly as designed.
Blackholed path — the hang: now put a firewall in front of the server that drops inbound ICMP. The server again sends 1500-byte DF segments; the tunnel again drops them and sends the ICMP 'fragmentation needed' — but the firewall discards it, so the server never hears back. The server's cached PMTU stays at 1500, so it retransmits the same oversized segments, which the tunnel drops again. The client, meanwhile, received the handshake but no body: the download stalls at zero or a few bytes and eventually times out. Every diagnostic that uses small packets (ping, a quick curl of a small page) works perfectly, which sends the operator hunting in the wrong place. This is the PMTU blackhole in its natural habitat.
The fixes bite: two changes rescue it. First, allow the ICMP 'fragmentation needed' message (type 3 code 4) through the firewall — it is not a threat, it is essential signaling — and discovery converges as in the healthy case. Second, and more robustly, clamp the MSS at the tunnel edge: a device on the tunnel rewrites the MSS option in the TCP handshake down to 1360 (1400 minus TCP/IP headers), so the server and client agree from the very first data segment to send pieces that fit the tunnel, and no oversized packet is ever emitted — the blackhole cannot form because there is nothing for the tunnel to reject. With the clamp in place the transfer runs full-speed and the operator never sees the hang at all — which is exactly why clamping sub-1500 tunnels is standard practice rather than relying on ICMP to always get through.